diff --git a/html/.DS_Store b/html/.DS_Store deleted file mode 100644 index c3f7922..0000000 Binary files a/html/.DS_Store and /dev/null differ diff --git a/html/C01EB6DCLHX-0.html b/html/C01EB6DCLHX-0.html deleted file mode 100644 index aaad435..0000000 --- a/html/C01EB6DCLHX-0.html +++ /dev/null @@ -1,6 +0,0 @@ -Slack

github-notifications

Created by Julien Le Dem on Tuesday, November 3rd, 2020

GitHubSunday, August 20th, 2023 at 12:17:54 AM GMT-04:00
UnknownSunday, September 3rd, 2023 at 12:14:43 AM GMT-04:00
GitHubWednesday, August 23rd, 2023 at 5:15:51 AM GMT-04:00
UnknownWednesday, August 23rd, 2023 at 10:53:27 AM GMT-04:00
GitHubWednesday, August 23rd, 2023 at 5:19:59 AM GMT-04:00
UnknownWednesday, August 23rd, 2023 at 5:19:59 AM GMT-04:00
UnknownWednesday, August 23rd, 2023 at 5:19:59 AM GMT-04:00
GitHubWednesday, August 23rd, 2023 at 5:22:03 AM GMT-04:00
GitHubWednesday, August 23rd, 2023 at 5:24:43 AM GMT-04:00
UnknownWednesday, August 23rd, 2023 at 10:53:27 AM GMT-04:00
UnknownWednesday, August 23rd, 2023 at 10:53:27 AM GMT-04:00
GitHubWednesday, August 23rd, 2023 at 10:53:27 AM GMT-04:00
GitHubWednesday, August 23rd, 2023 at 10:55:26 AM GMT-04:00
GitHubWednesday, August 23rd, 2023 at 10:58:15 AM GMT-04:00
GitHubWednesday, August 23rd, 2023 at 11:51:12 AM GMT-04:00
UnknownMonday, August 28th, 2023 at 7:54:18 PM GMT-04:00
GitHubWednesday, August 23rd, 2023 at 1:37:20 PM GMT-04:00
UnknownWednesday, August 23rd, 2023 at 2:35:08 PM GMT-04:00
GitHubWednesday, August 23rd, 2023 at 2:35:07 PM GMT-04:00
UnknownWednesday, August 23rd, 2023 at 2:35:08 PM GMT-04:00
GitHubWednesday, August 23rd, 2023 at 2:36:37 PM GMT-04:00
GitHubWednesday, August 23rd, 2023 at 2:37:13 PM GMT-04:00
GitHubWednesday, August 23rd, 2023 at 2:38:32 PM GMT-04:00
GitHubWednesday, August 23rd, 2023 at 2:40:03 PM GMT-04:00
GitHubWednesday, August 23rd, 2023 at 2:41:09 PM GMT-04:00
GitHubWednesday, August 23rd, 2023 at 3:00:09 PM GMT-04:00
GitHubThursday, August 24th, 2023 at 6:41:40 PM GMT-04:00
GitHubFriday, August 25th, 2023 at 9:04:33 AM GMT-04:00
UnknownFriday, August 25th, 2023 at 1:54:37 PM GMT-04:00
GitHubFriday, August 25th, 2023 at 11:38:00 AM GMT-04:00
UnknownFriday, September 1st, 2023 at 11:06:26 AM GMT-04:00
UnknownFriday, August 25th, 2023 at 1:54:37 PM GMT-04:00
GitHubFriday, August 25th, 2023 at 1:54:37 PM GMT-04:00
GitHubFriday, August 25th, 2023 at 1:56:45 PM GMT-04:00
GitHubSunday, August 27th, 2023 at 12:14:04 AM GMT-04:00
UnknownMonday, August 28th, 2023 at 2:55:46 AM GMT-04:00
GitHubSunday, August 27th, 2023 at 12:14:15 AM GMT-04:00
UnknownMonday, August 28th, 2023 at 3:09:54 AM GMT-04:00
GitHubSunday, August 27th, 2023 at 12:14:52 AM GMT-04:00
UnknownMonday, August 28th, 2023 at 8:41:27 AM GMT-04:00
GitHubSunday, August 27th, 2023 at 12:56:40 AM GMT-04:00
UnknownMonday, August 28th, 2023 at 2:56:11 AM GMT-04:00
GitHubSunday, August 27th, 2023 at 12:57:35 AM GMT-04:00
UnknownMonday, August 28th, 2023 at 2:55:16 AM GMT-04:00
GitHubSunday, August 27th, 2023 at 12:58:31 AM GMT-04:00
UnknownMonday, August 28th, 2023 at 8:36:21 AM GMT-04:00
GitHubSunday, August 27th, 2023 at 12:58:35 AM GMT-04:00
UnknownMonday, August 28th, 2023 at 3:11:23 AM GMT-04:00
GitHubMonday, August 28th, 2023 at 2:55:15 AM GMT-04:00
UnknownMonday, August 28th, 2023 at 2:55:16 AM GMT-04:00
GitHubMonday, August 28th, 2023 at 2:55:46 AM GMT-04:00
UnknownMonday, August 28th, 2023 at 2:55:46 AM GMT-04:00
UnknownMonday, August 28th, 2023 at 2:56:11 AM GMT-04:00
GitHubMonday, August 28th, 2023 at 2:57:10 AM GMT-04:00
GitHubMonday, August 28th, 2023 at 2:57:45 AM GMT-04:00
GitHubMonday, August 28th, 2023 at 3:00:09 AM GMT-04:00
GitHubMonday, August 28th, 2023 at 3:09:53 AM GMT-04:00
UnknownMonday, August 28th, 2023 at 3:09:54 AM GMT-04:00
GitHubMonday, August 28th, 2023 at 3:11:23 AM GMT-04:00
UnknownMonday, August 28th, 2023 at 3:11:23 AM GMT-04:00
GitHubMonday, August 28th, 2023 at 3:11:30 AM GMT-04:00
GitHubMonday, August 28th, 2023 at 3:13:29 AM GMT-04:00
GitHubMonday, August 28th, 2023 at 3:14:06 AM GMT-04:00
GitHubMonday, August 28th, 2023 at 3:15:37 AM GMT-04:00
UnknownMonday, August 28th, 2023 at 5:33:42 AM GMT-04:00
GitHubMonday, August 28th, 2023 at 7:47:29 AM GMT-04:00
UnknownMonday, August 28th, 2023 at 7:47:30 AM GMT-04:00
UnknownMonday, August 28th, 2023 at 8:39:00 AM GMT-04:00
UnknownMonday, August 28th, 2023 at 7:47:30 AM GMT-04:00
GitHubMonday, August 28th, 2023 at 8:36:21 AM GMT-04:00
UnknownMonday, August 28th, 2023 at 8:36:21 AM GMT-04:00
GitHubMonday, August 28th, 2023 at 8:38:08 AM GMT-04:00
GitHubMonday, August 28th, 2023 at 8:39:00 AM GMT-04:00
UnknownMonday, August 28th, 2023 at 8:39:00 AM GMT-04:00
UnknownMonday, August 28th, 2023 at 8:39:43 AM GMT-04:00
GitHubMonday, August 28th, 2023 at 8:40:29 AM GMT-04:00
GitHubMonday, August 28th, 2023 at 8:41:26 AM GMT-04:00
UnknownMonday, August 28th, 2023 at 8:41:27 AM GMT-04:00
GitHubMonday, August 28th, 2023 at 8:42:26 AM GMT-04:00
GitHubMonday, August 28th, 2023 at 8:43:36 AM GMT-04:00
GitHubMonday, August 28th, 2023 at 8:44:31 AM GMT-04:00
GitHubMonday, August 28th, 2023 at 8:46:33 AM GMT-04:00
GitHubMonday, August 28th, 2023 at 9:39:40 AM GMT-04:00
UnknownMonday, August 28th, 2023 at 10:22:42 AM GMT-04:00
GitHubMonday, August 28th, 2023 at 10:22:41 AM GMT-04:00
UnknownMonday, August 28th, 2023 at 10:22:42 AM GMT-04:00
GitHubMonday, August 28th, 2023 at 10:24:34 AM GMT-04:00
GitHubMonday, August 28th, 2023 at 10:27:03 AM GMT-04:00
GitHubMonday, August 28th, 2023 at 5:23:46 PM GMT-04:00
UnknownWednesday, September 6th, 2023 at 2:27:48 PM GMT-04:00
GitHubMonday, August 28th, 2023 at 7:54:17 PM GMT-04:00
UnknownMonday, August 28th, 2023 at 7:54:18 PM GMT-04:00
GitHubMonday, August 28th, 2023 at 7:57:07 PM GMT-04:00
GitHubTuesday, August 29th, 2023 at 1:45:20 AM GMT-04:00
UnknownWednesday, August 30th, 2023 at 4:18:06 PM GMT-04:00
GitHubWednesday, August 30th, 2023 at 6:22:00 AM GMT-04:00
GitHubWednesday, August 30th, 2023 at 8:16:30 AM GMT-04:00
UnknownWednesday, August 30th, 2023 at 8:35:01 AM GMT-04:00
GitHubWednesday, August 30th, 2023 at 8:35:01 AM GMT-04:00
UnknownWednesday, August 30th, 2023 at 8:35:01 AM GMT-04:00
GitHubWednesday, August 30th, 2023 at 8:37:32 AM GMT-04:00
GitHubWednesday, August 30th, 2023 at 9:07:52 AM GMT-04:00
UnknownWednesday, August 30th, 2023 at 9:07:52 AM GMT-04:00
UnknownMonday, September 4th, 2023 at 10:15:19 AM GMT-04:00
UnknownWednesday, August 30th, 2023 at 9:07:52 AM GMT-04:00
UnknownWednesday, August 30th, 2023 at 9:33:42 AM GMT-04:00
GitHubWednesday, August 30th, 2023 at 9:33:43 AM GMT-04:00
UnknownWednesday, August 30th, 2023 at 9:33:43 AM GMT-04:00
GitHubWednesday, August 30th, 2023 at 9:35:33 AM GMT-04:00
GitHubWednesday, August 30th, 2023 at 9:37:40 AM GMT-04:00
UnknownWednesday, August 30th, 2023 at 4:18:06 PM GMT-04:00
GitHubThursday, August 31st, 2023 at 4:36:21 PM GMT-04:00
UnknownFriday, September 1st, 2023 at 8:56:00 AM GMT-04:00
GitHubFriday, September 1st, 2023 at 8:55:59 AM GMT-04:00
UnknownFriday, September 1st, 2023 at 8:56:00 AM GMT-04:00
GitHubFriday, September 1st, 2023 at 8:59:19 AM GMT-04:00
GitHubFriday, September 1st, 2023 at 11:06:25 AM GMT-04:00
UnknownFriday, September 1st, 2023 at 11:06:26 AM GMT-04:00
GitHubFriday, September 1st, 2023 at 11:09:39 AM GMT-04:00
GitHubSaturday, September 2nd, 2023 at 6:19:52 AM GMT-04:00
UnknownSaturday, September 2nd, 2023 at 6:19:53 AM GMT-04:00
UnknownSaturday, September 2nd, 2023 at 6:19:53 AM GMT-04:00
GitHubSunday, September 3rd, 2023 at 12:14:35 AM GMT-04:00
UnknownMonday, September 4th, 2023 at 1:59:12 AM GMT-04:00
GitHubSunday, September 3rd, 2023 at 12:14:39 AM GMT-04:00
UnknownMonday, September 4th, 2023 at 2:01:23 AM GMT-04:00
UnknownSunday, September 3rd, 2023 at 12:14:43 AM GMT-04:00
GitHubSunday, September 3rd, 2023 at 12:25:46 AM GMT-04:00
UnknownMonday, September 4th, 2023 at 2:00:00 AM GMT-04:00
GitHubSunday, September 3rd, 2023 at 12:25:49 AM GMT-04:00
UnknownMonday, September 4th, 2023 at 1:58:46 AM GMT-04:00
GitHubSunday, September 3rd, 2023 at 12:55:45 AM GMT-04:00
UnknownMonday, September 4th, 2023 at 4:57:44 PM GMT-04:00
GitHubMonday, September 4th, 2023 at 1:58:45 AM GMT-04:00
UnknownMonday, September 4th, 2023 at 1:58:46 AM GMT-04:00
UnknownMonday, September 4th, 2023 at 1:59:12 AM GMT-04:00
GitHubMonday, September 4th, 2023 at 1:59:12 AM GMT-04:00
GitHubMonday, September 4th, 2023 at 1:59:59 AM GMT-04:00
UnknownMonday, September 4th, 2023 at 2:00:00 AM GMT-04:00
GitHubMonday, September 4th, 2023 at 2:00:41 AM GMT-04:00
GitHubMonday, September 4th, 2023 at 2:01:02 AM GMT-04:00
UnknownMonday, September 4th, 2023 at 2:01:23 AM GMT-04:00
GitHubMonday, September 4th, 2023 at 2:01:46 AM GMT-04:00
GitHubMonday, September 4th, 2023 at 2:03:45 AM GMT-04:00
GitHubMonday, September 4th, 2023 at 2:04:46 AM GMT-04:00
GitHubMonday, September 4th, 2023 at 6:35:02 AM GMT-04:00
GitHubMonday, September 4th, 2023 at 10:15:18 AM GMT-04:00
UnknownMonday, September 4th, 2023 at 10:15:19 AM GMT-04:00
UnknownMonday, September 4th, 2023 at 10:15:20 AM GMT-04:00
GitHubMonday, September 4th, 2023 at 10:16:56 AM GMT-04:00
GitHubMonday, September 4th, 2023 at 11:22:11 AM GMT-04:00
UnknownMonday, September 4th, 2023 at 11:22:12 AM GMT-04:00
GitHubMonday, September 4th, 2023 at 11:24:24 AM GMT-04:00
GitHubMonday, September 4th, 2023 at 11:26:40 AM GMT-04:00
GitHubMonday, September 4th, 2023 at 4:57:44 PM GMT-04:00
UnknownMonday, September 4th, 2023 at 4:57:44 PM GMT-04:00
GitHubMonday, September 4th, 2023 at 4:59:48 PM GMT-04:00
GitHubTuesday, September 5th, 2023 at 2:27:59 AM GMT-04:00
UnknownTuesday, September 5th, 2023 at 2:27:59 AM GMT-04:00
UnknownFriday, September 8th, 2023 at 9:14:58 AM GMT-04:00
UnknownTuesday, September 5th, 2023 at 2:27:59 AM GMT-04:00
GitHubTuesday, September 5th, 2023 at 3:52:57 AM GMT-04:00
GitHubTuesday, September 5th, 2023 at 4:26:27 AM GMT-04:00
GitHubTuesday, September 5th, 2023 at 9:59:11 AM GMT-04:00
UnknownTuesday, September 5th, 2023 at 2:24:40 PM GMT-04:00
GitHubTuesday, September 5th, 2023 at 11:10:34 AM GMT-04:00
UnknownTuesday, September 5th, 2023 at 2:08:32 PM GMT-04:00
GitHubTuesday, September 5th, 2023 at 2:08:31 PM GMT-04:00
UnknownTuesday, September 5th, 2023 at 2:08:32 PM GMT-04:00
GitHubTuesday, September 5th, 2023 at 2:10:51 PM GMT-04:00
GitHubTuesday, September 5th, 2023 at 2:24:40 PM GMT-04:00
UnknownTuesday, September 5th, 2023 at 2:24:40 PM GMT-04:00
GitHubTuesday, September 5th, 2023 at 2:26:57 PM GMT-04:00
GitHubWednesday, September 6th, 2023 at 4:46:14 AM GMT-04:00
UnknownWednesday, September 6th, 2023 at 4:46:14 AM GMT-04:00
GitHubWednesday, September 6th, 2023 at 4:48:07 AM GMT-04:00
GitHubWednesday, September 6th, 2023 at 4:50:34 AM GMT-04:00
GitHubWednesday, September 6th, 2023 at 5:23:31 AM GMT-04:00
UnknownWednesday, September 6th, 2023 at 5:23:31 AM GMT-04:00
UnknownWednesday, September 6th, 2023 at 5:23:31 AM GMT-04:00
GitHubWednesday, September 6th, 2023 at 2:27:47 PM GMT-04:00
UnknownWednesday, September 6th, 2023 at 2:27:48 PM GMT-04:00
GitHubWednesday, September 6th, 2023 at 2:29:58 PM GMT-04:00
GitHubWednesday, September 6th, 2023 at 4:16:46 PM GMT-04:00
UnknownWednesday, September 6th, 2023 at 4:31:06 PM GMT-04:00
UnknownWednesday, September 6th, 2023 at 4:31:06 PM GMT-04:00
GitHubWednesday, September 6th, 2023 at 4:44:05 PM GMT-04:00
GitHubThursday, September 7th, 2023 at 3:09:06 PM GMT-04:00
UnknownFriday, September 8th, 2023 at 7:20:04 AM GMT-04:00
GitHubFriday, September 8th, 2023 at 3:02:07 AM GMT-04:00
UnknownFriday, September 8th, 2023 at 3:02:07 AM GMT-04:00
UnknownMonday, September 11th, 2023 at 7:20:06 AM GMT-04:00
UnknownFriday, September 8th, 2023 at 3:02:07 AM GMT-04:00
GitHubFriday, September 8th, 2023 at 4:20:09 AM GMT-04:00
UnknownFriday, September 8th, 2023 at 6:18:48 AM GMT-04:00
GitHubFriday, September 8th, 2023 at 6:18:48 AM GMT-04:00
UnknownFriday, September 8th, 2023 at 6:18:48 AM GMT-04:00
GitHubFriday, September 8th, 2023 at 6:20:27 AM GMT-04:00
GitHubFriday, September 8th, 2023 at 6:22:40 AM GMT-04:00
GitHubFriday, September 8th, 2023 at 7:20:04 AM GMT-04:00
UnknownFriday, September 8th, 2023 at 7:20:04 AM GMT-04:00
GitHubFriday, September 8th, 2023 at 7:21:59 AM GMT-04:00
GitHubFriday, September 8th, 2023 at 7:24:10 AM GMT-04:00
GitHubFriday, September 8th, 2023 at 9:14:58 AM GMT-04:00
UnknownFriday, September 8th, 2023 at 9:14:58 AM GMT-04:00
UnknownFriday, September 8th, 2023 at 9:14:58 AM GMT-04:00
GitHubFriday, September 8th, 2023 at 9:16:53 AM GMT-04:00
GitHubFriday, September 8th, 2023 at 9:19:20 AM GMT-04:00
GitHubFriday, September 8th, 2023 at 9:23:26 AM GMT-04:00
UnknownFriday, September 8th, 2023 at 1:27:13 PM GMT-04:00
GitHubFriday, September 8th, 2023 at 1:27:13 PM GMT-04:00
UnknownFriday, September 8th, 2023 at 1:27:13 PM GMT-04:00
GitHubFriday, September 8th, 2023 at 1:29:11 PM GMT-04:00
GitHubFriday, September 8th, 2023 at 1:31:53 PM GMT-04:00
GitHubFriday, September 8th, 2023 at 4:34:36 PM GMT-04:00
UnknownThursday, September 14th, 2023 at 11:02:48 AM GMT-04:00
GitHubSunday, September 10th, 2023 at 12:13:54 AM GMT-04:00
UnknownThursday, September 28th, 2023 at 4:25:39 AM GMT-04:00
GitHubSunday, September 10th, 2023 at 12:32:47 AM GMT-04:00
UnknownMonday, September 11th, 2023 at 2:14:33 AM GMT-04:00
GitHubSunday, September 10th, 2023 at 12:32:48 AM GMT-04:00
UnknownMonday, September 11th, 2023 at 2:15:06 AM GMT-04:00
GitHubSunday, September 10th, 2023 at 12:38:44 AM GMT-04:00
UnknownTuesday, September 12th, 2023 at 7:55:00 AM GMT-04:00
UnknownMonday, September 11th, 2023 at 2:14:33 AM GMT-04:00
GitHubMonday, September 11th, 2023 at 2:15:06 AM GMT-04:00
UnknownMonday, September 11th, 2023 at 2:15:06 AM GMT-04:00
GitHubMonday, September 11th, 2023 at 2:17:05 AM GMT-04:00
GitHubMonday, September 11th, 2023 at 2:19:05 AM GMT-04:00
GitHubMonday, September 11th, 2023 at 4:30:26 AM GMT-04:00
UnknownTuesday, September 12th, 2023 at 6:59:17 AM GMT-04:00
GitHubMonday, September 11th, 2023 at 5:12:27 AM GMT-04:00
UnknownTuesday, September 12th, 2023 at 6:59:17 AM GMT-04:00
GitHubMonday, September 11th, 2023 at 7:20:04 AM GMT-04:00
GitHubMonday, September 11th, 2023 at 7:20:05 AM GMT-04:00
UnknownMonday, September 11th, 2023 at 7:20:05 AM GMT-04:00
UnknownMonday, September 11th, 2023 at 7:20:05 AM GMT-04:00
UnknownMonday, September 11th, 2023 at 7:20:06 AM GMT-04:00
GitHubMonday, September 11th, 2023 at 7:21:50 AM GMT-04:00
GitHubMonday, September 11th, 2023 at 7:24:06 AM GMT-04:00
GitHubMonday, September 11th, 2023 at 8:52:20 AM GMT-04:00
UnknownMonday, September 11th, 2023 at 8:52:21 AM GMT-04:00
UnknownMonday, September 11th, 2023 at 8:52:21 AM GMT-04:00
GitHubMonday, September 11th, 2023 at 5:35:24 PM GMT-04:00
UnknownTuesday, September 12th, 2023 at 9:02:38 AM GMT-04:00
GitHubTuesday, September 12th, 2023 at 5:17:14 AM GMT-04:00
UnknownTuesday, September 12th, 2023 at 5:17:15 AM GMT-04:00
UnknownTuesday, September 12th, 2023 at 5:23:09 AM GMT-04:00
UnknownTuesday, September 12th, 2023 at 5:17:15 AM GMT-04:00
GitHubTuesday, September 12th, 2023 at 5:23:09 AM GMT-04:00
UnknownTuesday, September 12th, 2023 at 5:23:09 AM GMT-04:00
GitHubTuesday, September 12th, 2023 at 5:25:02 AM GMT-04:00
GitHubTuesday, September 12th, 2023 at 5:27:22 AM GMT-04:00
GitHubTuesday, September 12th, 2023 at 6:59:17 AM GMT-04:00
UnknownTuesday, September 12th, 2023 at 6:59:17 AM GMT-04:00
UnknownTuesday, September 12th, 2023 at 6:59:17 AM GMT-04:00
GitHubTuesday, September 12th, 2023 at 7:01:15 AM GMT-04:00
GitHubTuesday, September 12th, 2023 at 7:03:22 AM GMT-04:00
UnknownTuesday, September 12th, 2023 at 7:55:00 AM GMT-04:00
GitHubTuesday, September 12th, 2023 at 9:02:38 AM GMT-04:00
UnknownTuesday, September 12th, 2023 at 9:02:38 AM GMT-04:00
GitHubTuesday, September 12th, 2023 at 9:04:33 AM GMT-04:00
GitHubTuesday, September 12th, 2023 at 9:06:19 AM GMT-04:00
UnknownTuesday, September 12th, 2023 at 9:19:54 AM GMT-04:00
GitHubTuesday, September 12th, 2023 at 9:07:38 AM GMT-04:00
GitHubTuesday, September 12th, 2023 at 9:19:53 AM GMT-04:00
UnknownTuesday, September 12th, 2023 at 9:19:54 AM GMT-04:00
GitHubTuesday, September 12th, 2023 at 9:22:16 AM GMT-04:00
GitHubTuesday, September 12th, 2023 at 5:24:34 PM GMT-04:00
UnknownWednesday, September 13th, 2023 at 12:29:19 PM GMT-04:00
GitHubWednesday, September 13th, 2023 at 2:21:59 AM GMT-04:00
UnknownWednesday, September 13th, 2023 at 2:22:00 AM GMT-04:00
UnknownWednesday, September 13th, 2023 at 5:15:25 AM GMT-04:00
UnknownWednesday, September 13th, 2023 at 2:22:00 AM GMT-04:00
GitHubWednesday, September 13th, 2023 at 5:15:25 AM GMT-04:00
UnknownWednesday, September 13th, 2023 at 5:15:25 AM GMT-04:00
GitHubWednesday, September 13th, 2023 at 5:17:46 AM GMT-04:00
GitHubWednesday, September 13th, 2023 at 5:20:10 AM GMT-04:00
GitHubWednesday, September 13th, 2023 at 7:54:43 AM GMT-04:00
GitHubWednesday, September 13th, 2023 at 9:07:30 AM GMT-04:00
UnknownWednesday, September 13th, 2023 at 10:35:09 AM GMT-04:00
GitHubWednesday, September 13th, 2023 at 10:35:08 AM GMT-04:00
UnknownWednesday, September 13th, 2023 at 10:35:09 AM GMT-04:00
GitHubWednesday, September 13th, 2023 at 10:38:06 AM GMT-04:00
GitHubWednesday, September 13th, 2023 at 12:29:19 PM GMT-04:00
UnknownWednesday, September 13th, 2023 at 12:29:19 PM GMT-04:00
GitHubWednesday, September 13th, 2023 at 12:31:45 PM GMT-04:00
GitHubWednesday, September 13th, 2023 at 3:13:43 PM GMT-04:00
UnknownTuesday, September 26th, 2023 at 5:42:01 PM GMT-04:00
GitHubWednesday, September 13th, 2023 at 3:25:55 PM GMT-04:00
GitHubThursday, September 14th, 2023 at 7:10:43 AM GMT-04:00
UnknownThursday, September 14th, 2023 at 11:02:48 AM GMT-04:00
GitHubThursday, September 14th, 2023 at 8:27:36 AM GMT-04:00
GitHubThursday, September 14th, 2023 at 9:26:32 AM GMT-04:00
UnknownThursday, September 14th, 2023 at 11:03:43 AM GMT-04:00
GitHubThursday, September 14th, 2023 at 9:54:31 AM GMT-04:00
GitHubThursday, September 14th, 2023 at 11:02:47 AM GMT-04:00
UnknownThursday, September 14th, 2023 at 11:02:48 AM GMT-04:00
UnknownThursday, September 14th, 2023 at 11:02:48 AM GMT-04:00
GitHubThursday, September 14th, 2023 at 11:03:42 AM GMT-04:00
UnknownThursday, September 14th, 2023 at 11:03:43 AM GMT-04:00
GitHubThursday, September 14th, 2023 at 11:04:51 AM GMT-04:00
GitHubThursday, September 14th, 2023 at 11:05:37 AM GMT-04:00
GitHubThursday, September 14th, 2023 at 11:07:09 AM GMT-04:00
GitHubThursday, September 14th, 2023 at 11:07:56 AM GMT-04:00
GitHubThursday, September 14th, 2023 at 11:29:49 AM GMT-04:00
GitHubThursday, September 14th, 2023 at 11:56:53 AM GMT-04:00
UnknownThursday, September 14th, 2023 at 12:18:21 PM GMT-04:00
GitHubThursday, September 14th, 2023 at 12:18:20 PM GMT-04:00
UnknownThursday, September 14th, 2023 at 12:18:21 PM GMT-04:00
GitHubThursday, September 14th, 2023 at 12:20:13 PM GMT-04:00
GitHubThursday, September 14th, 2023 at 12:22:54 PM GMT-04:00
GitHubThursday, September 14th, 2023 at 12:27:32 PM GMT-04:00
UnknownThursday, September 14th, 2023 at 12:32:38 PM GMT-04:00
GitHubThursday, September 14th, 2023 at 12:32:38 PM GMT-04:00
UnknownThursday, September 14th, 2023 at 12:32:38 PM GMT-04:00
GitHubThursday, September 14th, 2023 at 12:34:00 PM GMT-04:00
GitHubThursday, September 14th, 2023 at 12:34:23 PM GMT-04:00
GitHubThursday, September 14th, 2023 at 12:35:58 PM GMT-04:00
GitHubThursday, September 14th, 2023 at 12:37:10 PM GMT-04:00
GitHubThursday, September 14th, 2023 at 12:38:21 PM GMT-04:00
GitHubThursday, September 14th, 2023 at 2:22:05 PM GMT-04:00
UnknownTuesday, September 19th, 2023 at 6:22:33 PM GMT-04:00
GitHubFriday, September 15th, 2023 at 2:36:54 AM GMT-04:00
UnknownFriday, September 15th, 2023 at 2:36:54 AM GMT-04:00
UnknownFriday, September 15th, 2023 at 4:38:27 AM GMT-04:00
UnknownFriday, September 15th, 2023 at 2:36:54 AM GMT-04:00
GitHubFriday, September 15th, 2023 at 4:38:27 AM GMT-04:00
UnknownFriday, September 15th, 2023 at 4:38:27 AM GMT-04:00
GitHubFriday, September 15th, 2023 at 4:40:21 AM GMT-04:00
GitHubFriday, September 15th, 2023 at 4:43:08 AM GMT-04:00
GitHubFriday, September 15th, 2023 at 9:45:13 AM GMT-04:00
UnknownFriday, September 15th, 2023 at 9:53:15 AM GMT-04:00
GitHubFriday, September 15th, 2023 at 9:53:14 AM GMT-04:00
UnknownFriday, September 15th, 2023 at 9:53:15 AM GMT-04:00
GitHubFriday, September 15th, 2023 at 9:55:32 AM GMT-04:00
GitHubSunday, September 17th, 2023 at 12:20:03 AM GMT-04:00
UnknownWednesday, September 20th, 2023 at 10:57:05 AM GMT-04:00
GitHubSunday, September 17th, 2023 at 12:20:08 AM GMT-04:00
UnknownTuesday, September 26th, 2023 at 8:41:23 AM GMT-04:00
GitHubSunday, September 17th, 2023 at 12:46:03 AM GMT-04:00
UnknownSunday, September 24th, 2023 at 7:42:33 PM GMT-04:00
GitHubTuesday, September 19th, 2023 at 12:24:52 PM GMT-04:00
UnknownTuesday, September 19th, 2023 at 2:15:15 PM GMT-04:00
GitHubTuesday, September 19th, 2023 at 2:15:15 PM GMT-04:00
UnknownTuesday, September 19th, 2023 at 2:15:15 PM GMT-04:00
GitHubTuesday, September 19th, 2023 at 2:17:35 PM GMT-04:00
GitHubTuesday, September 19th, 2023 at 5:42:21 PM GMT-04:00
UnknownTuesday, September 19th, 2023 at 5:52:33 PM GMT-04:00
GitHubTuesday, September 19th, 2023 at 5:52:33 PM GMT-04:00
UnknownTuesday, September 19th, 2023 at 5:52:33 PM GMT-04:00
GitHubTuesday, September 19th, 2023 at 5:54:29 PM GMT-04:00
GitHubTuesday, September 19th, 2023 at 5:54:31 PM GMT-04:00
GitHubTuesday, September 19th, 2023 at 5:56:30 PM GMT-04:00
GitHubTuesday, September 19th, 2023 at 5:57:15 PM GMT-04:00
GitHubTuesday, September 19th, 2023 at 5:58:45 PM GMT-04:00
GitHubTuesday, September 19th, 2023 at 6:22:32 PM GMT-04:00
UnknownTuesday, September 19th, 2023 at 6:22:33 PM GMT-04:00
GitHubTuesday, September 19th, 2023 at 6:24:23 PM GMT-04:00
GitHubTuesday, September 19th, 2023 at 6:27:07 PM GMT-04:00
GitHubTuesday, September 19th, 2023 at 6:27:26 PM GMT-04:00
UnknownTuesday, September 19th, 2023 at 6:36:38 PM GMT-04:00
GitHubTuesday, September 19th, 2023 at 6:36:37 PM GMT-04:00
UnknownTuesday, September 19th, 2023 at 6:36:38 PM GMT-04:00
GitHubTuesday, September 19th, 2023 at 6:38:40 PM GMT-04:00
GitHubTuesday, September 19th, 2023 at 6:40:47 PM GMT-04:00
UnknownWednesday, September 20th, 2023 at 10:57:05 AM GMT-04:00
GitHubWednesday, September 20th, 2023 at 11:28:22 AM GMT-04:00
GitHubWednesday, September 20th, 2023 at 11:30:09 AM GMT-04:00
GitHubWednesday, September 20th, 2023 at 11:32:28 AM GMT-04:00
GitHubWednesday, September 20th, 2023 at 2:56:40 PM GMT-04:00
GitHubWednesday, September 20th, 2023 at 4:09:10 PM GMT-04:00
GitHubFriday, September 22nd, 2023 at 5:34:53 AM GMT-04:00
UnknownFriday, September 22nd, 2023 at 7:01:25 AM GMT-04:00
GitHubFriday, September 22nd, 2023 at 7:01:24 AM GMT-04:00
UnknownFriday, September 22nd, 2023 at 7:01:25 AM GMT-04:00
GitHubFriday, September 22nd, 2023 at 7:03:27 AM GMT-04:00
GitHubFriday, September 22nd, 2023 at 7:05:45 AM GMT-04:00
GitHubSaturday, September 23rd, 2023 at 3:55:04 PM GMT-04:00
GitHubSunday, September 24th, 2023 at 12:10:59 AM GMT-04:00
UnknownTuesday, September 26th, 2023 at 2:38:06 AM GMT-04:00
GitHubSunday, September 24th, 2023 at 12:17:59 AM GMT-04:00
UnknownTuesday, September 26th, 2023 at 2:41:10 AM GMT-04:00
GitHubSunday, September 24th, 2023 at 12:18:06 AM GMT-04:00
UnknownTuesday, September 26th, 2023 at 2:41:45 AM GMT-04:00
GitHubSunday, September 24th, 2023 at 12:24:10 AM GMT-04:00
UnknownTuesday, September 26th, 2023 at 2:38:31 AM GMT-04:00
UnknownSunday, September 24th, 2023 at 7:42:33 PM GMT-04:00
GitHubMonday, September 25th, 2023 at 11:01:11 AM GMT-04:00
UnknownThursday, September 28th, 2023 at 7:50:07 AM GMT-04:00
GitHubMonday, September 25th, 2023 at 11:14:17 AM GMT-04:00
UnknownThursday, September 28th, 2023 at 7:51:57 AM GMT-04:00
GitHubTuesday, September 26th, 2023 at 2:24:32 AM GMT-04:00
UnknownWednesday, October 25th, 2023 at 3:34:22 AM GMT-04:00
GitHubTuesday, September 26th, 2023 at 2:38:06 AM GMT-04:00
UnknownTuesday, September 26th, 2023 at 2:38:06 AM GMT-04:00
UnknownTuesday, September 26th, 2023 at 2:38:31 AM GMT-04:00
GitHubTuesday, September 26th, 2023 at 2:38:31 AM GMT-04:00
GitHubTuesday, September 26th, 2023 at 2:40:06 AM GMT-04:00
GitHubTuesday, September 26th, 2023 at 2:40:20 AM GMT-04:00
GitHubTuesday, September 26th, 2023 at 2:41:08 AM GMT-04:00
UnknownTuesday, September 26th, 2023 at 2:41:10 AM GMT-04:00
GitHubTuesday, September 26th, 2023 at 2:41:45 AM GMT-04:00
UnknownTuesday, September 26th, 2023 at 2:41:45 AM GMT-04:00
GitHubTuesday, September 26th, 2023 at 2:42:35 AM GMT-04:00
GitHubTuesday, September 26th, 2023 at 2:44:09 AM GMT-04:00
GitHubTuesday, September 26th, 2023 at 2:44:49 AM GMT-04:00
GitHubTuesday, September 26th, 2023 at 2:46:24 AM GMT-04:00
GitHubTuesday, September 26th, 2023 at 2:47:01 AM GMT-04:00
GitHubTuesday, September 26th, 2023 at 7:22:25 AM GMT-04:00
UnknownTuesday, September 26th, 2023 at 7:22:26 AM GMT-04:00
UnknownTuesday, September 26th, 2023 at 7:40:05 AM GMT-04:00
UnknownTuesday, September 26th, 2023 at 7:22:26 AM GMT-04:00
GitHubTuesday, September 26th, 2023 at 7:40:04 AM GMT-04:00
UnknownTuesday, September 26th, 2023 at 7:40:05 AM GMT-04:00
GitHubTuesday, September 26th, 2023 at 7:42:03 AM GMT-04:00
GitHubTuesday, September 26th, 2023 at 7:44:19 AM GMT-04:00
GitHubTuesday, September 26th, 2023 at 8:00:33 AM GMT-04:00
UnknownTuesday, September 26th, 2023 at 8:00:33 AM GMT-04:00
UnknownTuesday, September 26th, 2023 at 8:00:33 AM GMT-04:00
GitHubTuesday, September 26th, 2023 at 8:39:46 AM GMT-04:00
UnknownTuesday, September 26th, 2023 at 8:39:47 AM GMT-04:00
UnknownTuesday, September 26th, 2023 at 8:39:47 AM GMT-04:00
GitHubTuesday, September 26th, 2023 at 8:41:23 AM GMT-04:00
UnknownTuesday, September 26th, 2023 at 8:41:23 AM GMT-04:00
GitHubTuesday, September 26th, 2023 at 8:43:12 AM GMT-04:00
GitHubTuesday, September 26th, 2023 at 8:45:30 AM GMT-04:00
GitHubTuesday, September 26th, 2023 at 5:41:59 PM GMT-04:00
UnknownTuesday, September 26th, 2023 at 5:42:01 PM GMT-04:00
GitHubTuesday, September 26th, 2023 at 5:44:04 PM GMT-04:00
GitHubWednesday, September 27th, 2023 at 3:07:11 AM GMT-04:00
GitHubWednesday, September 27th, 2023 at 8:19:41 AM GMT-04:00
GitHubWednesday, September 27th, 2023 at 8:19:47 AM GMT-04:00
UnknownWednesday, September 27th, 2023 at 8:19:47 AM GMT-04:00
UnknownWednesday, September 27th, 2023 at 8:19:47 AM GMT-04:00
GitHubWednesday, September 27th, 2023 at 8:21:50 AM GMT-04:00
GitHubWednesday, September 27th, 2023 at 8:24:15 AM GMT-04:00
GitHubWednesday, September 27th, 2023 at 8:32:12 AM GMT-04:00
UnknownWednesday, September 27th, 2023 at 8:32:13 AM GMT-04:00
UnknownMonday, October 16th, 2023 at 4:30:59 AM GMT-04:00
UnknownWednesday, September 27th, 2023 at 8:32:13 AM GMT-04:00
GitHubWednesday, September 27th, 2023 at 9:02:53 AM GMT-04:00
UnknownFriday, October 6th, 2023 at 7:33:42 AM GMT-04:00
GitHubWednesday, September 27th, 2023 at 10:58:54 AM GMT-04:00
UnknownThursday, September 28th, 2023 at 7:50:05 AM GMT-04:00
GitHubWednesday, September 27th, 2023 at 12:43:15 PM GMT-04:00
GitHubThursday, September 28th, 2023 at 4:25:39 AM GMT-04:00
UnknownThursday, September 28th, 2023 at 4:25:39 AM GMT-04:00
GitHubThursday, September 28th, 2023 at 4:27:32 AM GMT-04:00
GitHubThursday, September 28th, 2023 at 4:30:01 AM GMT-04:00
GitHubThursday, September 28th, 2023 at 4:42:07 AM GMT-04:00
UnknownThursday, September 28th, 2023 at 7:51:57 AM GMT-04:00
GitHubThursday, September 28th, 2023 at 7:38:48 AM GMT-04:00
UnknownFriday, September 29th, 2023 at 9:20:27 AM GMT-04:00
UnknownThursday, September 28th, 2023 at 7:50:05 AM GMT-04:00
GitHubThursday, September 28th, 2023 at 7:50:06 AM GMT-04:00
UnknownThursday, September 28th, 2023 at 7:50:07 AM GMT-04:00
GitHubThursday, September 28th, 2023 at 7:51:56 AM GMT-04:00
UnknownThursday, September 28th, 2023 at 7:51:57 AM GMT-04:00
GitHubThursday, September 28th, 2023 at 7:51:57 AM GMT-04:00
UnknownThursday, September 28th, 2023 at 7:51:57 AM GMT-04:00
GitHubThursday, September 28th, 2023 at 7:53:40 AM GMT-04:00
GitHubThursday, September 28th, 2023 at 7:54:18 AM GMT-04:00
GitHubThursday, September 28th, 2023 at 7:56:35 AM GMT-04:00
GitHubThursday, September 28th, 2023 at 11:38:12 AM GMT-04:00
UnknownThursday, September 28th, 2023 at 12:02:27 PM GMT-04:00
UnknownThursday, September 28th, 2023 at 12:02:27 PM GMT-04:00
GitHubThursday, September 28th, 2023 at 12:02:27 PM GMT-04:00
GitHubThursday, September 28th, 2023 at 12:04:29 PM GMT-04:00
GitHubThursday, September 28th, 2023 at 12:06:52 PM GMT-04:00
GitHubThursday, September 28th, 2023 at 6:13:11 PM GMT-04:00
UnknownFriday, September 29th, 2023 at 4:43:53 AM GMT-04:00
GitHubFriday, September 29th, 2023 at 4:43:52 AM GMT-04:00
UnknownFriday, September 29th, 2023 at 4:43:53 AM GMT-04:00
GitHubFriday, September 29th, 2023 at 4:45:51 AM GMT-04:00
GitHubFriday, September 29th, 2023 at 4:48:17 AM GMT-04:00
GitHubFriday, September 29th, 2023 at 5:12:10 AM GMT-04:00
UnknownMonday, October 2nd, 2023 at 2:13:19 AM GMT-04:00
GitHubFriday, September 29th, 2023 at 7:11:10 AM GMT-04:00
UnknownFriday, September 29th, 2023 at 7:11:11 AM GMT-04:00
UnknownMonday, October 2nd, 2023 at 4:21:45 AM GMT-04:00
UnknownFriday, September 29th, 2023 at 7:11:11 AM GMT-04:00
GitHubFriday, September 29th, 2023 at 9:20:26 AM GMT-04:00
UnknownFriday, September 29th, 2023 at 9:20:27 AM GMT-04:00
GitHubFriday, September 29th, 2023 at 9:22:21 AM GMT-04:00
GitHubFriday, September 29th, 2023 at 9:25:29 AM GMT-04:00
GitHubFriday, September 29th, 2023 at 10:23:18 AM GMT-04:00
GitHubSunday, October 1st, 2023 at 12:34:14 AM GMT-04:00
UnknownMonday, October 2nd, 2023 at 2:14:36 AM GMT-04:00
GitHubSunday, October 1st, 2023 at 12:34:25 AM GMT-04:00
UnknownMonday, October 2nd, 2023 at 2:14:11 AM GMT-04:00
GitHubSunday, October 1st, 2023 at 12:51:58 AM GMT-04:00
UnknownSunday, October 1st, 2023 at 1:36:56 AM GMT-04:00
UnknownSunday, October 1st, 2023 at 1:36:56 AM GMT-04:00
GitHubSunday, October 1st, 2023 at 1:36:57 AM GMT-04:00
GitHubSunday, October 1st, 2023 at 1:38:43 AM GMT-04:00
GitHubSunday, October 1st, 2023 at 1:41:28 AM GMT-04:00
GitHubMonday, October 2nd, 2023 at 2:13:19 AM GMT-04:00
UnknownMonday, October 2nd, 2023 at 2:13:19 AM GMT-04:00
GitHubMonday, October 2nd, 2023 at 2:14:10 AM GMT-04:00
UnknownMonday, October 2nd, 2023 at 2:14:11 AM GMT-04:00
GitHubMonday, October 2nd, 2023 at 2:14:36 AM GMT-04:00
UnknownMonday, October 2nd, 2023 at 2:14:36 AM GMT-04:00
GitHubMonday, October 2nd, 2023 at 2:15:02 AM GMT-04:00
GitHubMonday, October 2nd, 2023 at 2:15:50 AM GMT-04:00
GitHubMonday, October 2nd, 2023 at 2:16:24 AM GMT-04:00
GitHubMonday, October 2nd, 2023 at 2:17:44 AM GMT-04:00
GitHubMonday, October 2nd, 2023 at 2:18:49 AM GMT-04:00
GitHubMonday, October 2nd, 2023 at 3:05:39 AM GMT-04:00
UnknownMonday, October 2nd, 2023 at 3:05:39 AM GMT-04:00
UnknownMonday, October 2nd, 2023 at 3:05:39 AM GMT-04:00
GitHubMonday, October 2nd, 2023 at 4:21:45 AM GMT-04:00
UnknownMonday, October 2nd, 2023 at 4:21:45 AM GMT-04:00
GitHubMonday, October 2nd, 2023 at 4:23:34 AM GMT-04:00
GitHubMonday, October 2nd, 2023 at 4:25:50 AM GMT-04:00
GitHubMonday, October 2nd, 2023 at 11:26:07 AM GMT-04:00
UnknownMonday, October 2nd, 2023 at 1:18:28 PM GMT-04:00
GitHubMonday, October 2nd, 2023 at 1:18:27 PM GMT-04:00
UnknownMonday, October 2nd, 2023 at 1:18:28 PM GMT-04:00
GitHubMonday, October 2nd, 2023 at 1:20:49 PM GMT-04:00
GitHubMonday, October 2nd, 2023 at 1:57:37 PM GMT-04:00
UnknownThursday, October 5th, 2023 at 6:52:49 AM GMT-04:00
GitHubMonday, October 2nd, 2023 at 2:37:37 PM GMT-04:00
UnknownTuesday, October 3rd, 2023 at 9:02:00 AM GMT-04:00
GitHubMonday, October 2nd, 2023 at 7:15:48 PM GMT-04:00
UnknownMonday, October 2nd, 2023 at 7:19:51 PM GMT-04:00
GitHubMonday, October 2nd, 2023 at 7:19:50 PM GMT-04:00
UnknownMonday, October 2nd, 2023 at 7:19:51 PM GMT-04:00
GitHubMonday, October 2nd, 2023 at 7:22:16 PM GMT-04:00
GitHubMonday, October 2nd, 2023 at 9:48:34 PM GMT-04:00
UnknownWednesday, October 4th, 2023 at 2:16:58 AM GMT-04:00
GitHubTuesday, October 3rd, 2023 at 8:30:35 AM GMT-04:00
GitHubTuesday, October 3rd, 2023 at 8:49:45 AM GMT-04:00
UnknownWednesday, October 4th, 2023 at 3:33:43 AM GMT-04:00
GitHubTuesday, October 3rd, 2023 at 9:02:00 AM GMT-04:00
UnknownTuesday, October 3rd, 2023 at 9:02:00 AM GMT-04:00
GitHubTuesday, October 3rd, 2023 at 9:03:12 AM GMT-04:00
GitHubTuesday, October 3rd, 2023 at 9:04:10 AM GMT-04:00
GitHubTuesday, October 3rd, 2023 at 9:05:11 AM GMT-04:00
GitHubTuesday, October 3rd, 2023 at 9:06:39 AM GMT-04:00
GitHubTuesday, October 3rd, 2023 at 9:07:42 AM GMT-04:00
GitHubTuesday, October 3rd, 2023 at 9:13:01 AM GMT-04:00
UnknownTuesday, October 3rd, 2023 at 10:07:58 AM GMT-04:00
GitHubTuesday, October 3rd, 2023 at 10:07:57 AM GMT-04:00
UnknownTuesday, October 3rd, 2023 at 10:07:58 AM GMT-04:00
GitHubTuesday, October 3rd, 2023 at 10:08:47 AM GMT-04:00
UnknownWednesday, October 4th, 2023 at 3:17:15 AM GMT-04:00
GitHubTuesday, October 3rd, 2023 at 10:08:49 AM GMT-04:00
UnknownWednesday, October 4th, 2023 at 2:41:11 AM GMT-04:00
GitHubTuesday, October 3rd, 2023 at 10:08:52 AM GMT-04:00
UnknownThursday, October 5th, 2023 at 4:13:10 PM GMT-04:00
GitHubTuesday, October 3rd, 2023 at 10:08:59 AM GMT-04:00
UnknownWednesday, October 4th, 2023 at 4:30:56 AM GMT-04:00
GitHubTuesday, October 3rd, 2023 at 10:09:15 AM GMT-04:00
GitHubTuesday, October 3rd, 2023 at 10:09:24 AM GMT-04:00
UnknownWednesday, October 4th, 2023 at 2:43:39 AM GMT-04:00
GitHubTuesday, October 3rd, 2023 at 10:09:30 AM GMT-04:00
UnknownWednesday, October 4th, 2023 at 2:43:00 AM GMT-04:00
GitHubTuesday, October 3rd, 2023 at 10:09:44 AM GMT-04:00
GitHubTuesday, October 3rd, 2023 at 10:11:59 AM GMT-04:00
GitHubTuesday, October 3rd, 2023 at 10:12:06 AM GMT-04:00
GitHubTuesday, October 3rd, 2023 at 10:14:38 AM GMT-04:00
GitHubTuesday, October 3rd, 2023 at 10:33:29 AM GMT-04:00
GitHubTuesday, October 3rd, 2023 at 2:58:00 PM GMT-04:00
UnknownWednesday, October 4th, 2023 at 11:59:14 AM GMT-04:00
UnknownWednesday, October 4th, 2023 at 2:16:58 AM GMT-04:00
GitHubWednesday, October 4th, 2023 at 2:16:58 AM GMT-04:00
GitHubWednesday, October 4th, 2023 at 2:18:47 AM GMT-04:00
GitHubWednesday, October 4th, 2023 at 2:21:05 AM GMT-04:00
GitHubWednesday, October 4th, 2023 at 2:41:10 AM GMT-04:00
UnknownWednesday, October 4th, 2023 at 2:41:11 AM GMT-04:00
GitHubWednesday, October 4th, 2023 at 2:42:57 AM GMT-04:00
UnknownWednesday, October 4th, 2023 at 2:43:00 AM GMT-04:00
GitHubWednesday, October 4th, 2023 at 2:43:38 AM GMT-04:00
UnknownWednesday, October 4th, 2023 at 2:43:39 AM GMT-04:00
GitHubWednesday, October 4th, 2023 at 2:45:37 AM GMT-04:00
GitHubWednesday, October 4th, 2023 at 2:45:46 AM GMT-04:00
GitHubWednesday, October 4th, 2023 at 2:48:00 AM GMT-04:00
UnknownWednesday, October 4th, 2023 at 3:17:15 AM GMT-04:00
GitHubWednesday, October 4th, 2023 at 3:33:42 AM GMT-04:00
UnknownWednesday, October 4th, 2023 at 3:33:43 AM GMT-04:00
GitHubWednesday, October 4th, 2023 at 3:35:46 AM GMT-04:00
GitHubWednesday, October 4th, 2023 at 3:38:48 AM GMT-04:00
UnknownWednesday, October 4th, 2023 at 4:30:56 AM GMT-04:00
GitHubWednesday, October 4th, 2023 at 4:30:57 AM GMT-04:00
GitHubWednesday, October 4th, 2023 at 4:32:48 AM GMT-04:00
GitHubWednesday, October 4th, 2023 at 4:35:12 AM GMT-04:00
GitHubWednesday, October 4th, 2023 at 8:48:47 AM GMT-04:00
UnknownMonday, October 9th, 2023 at 3:31:35 AM GMT-04:00
GitHubWednesday, October 4th, 2023 at 8:48:58 AM GMT-04:00
UnknownWednesday, October 4th, 2023 at 8:48:59 AM GMT-04:00
UnknownFriday, October 6th, 2023 at 7:33:42 AM GMT-04:00
UnknownWednesday, October 4th, 2023 at 8:48:59 AM GMT-04:00
GitHubWednesday, October 4th, 2023 at 9:15:04 AM GMT-04:00
UnknownWednesday, October 4th, 2023 at 5:59:23 PM GMT-04:00
GitHubWednesday, October 4th, 2023 at 10:57:04 AM GMT-04:00
UnknownWednesday, October 4th, 2023 at 11:59:14 AM GMT-04:00
GitHubWednesday, October 4th, 2023 at 11:59:14 AM GMT-04:00
GitHubWednesday, October 4th, 2023 at 12:02:15 PM GMT-04:00
UnknownWednesday, October 4th, 2023 at 5:59:23 PM GMT-04:00
GitHubWednesday, October 4th, 2023 at 5:59:23 PM GMT-04:00
GitHubWednesday, October 4th, 2023 at 6:06:38 PM GMT-04:00
GitHubWednesday, October 4th, 2023 at 8:24:20 PM GMT-04:00
GitHubThursday, October 5th, 2023 at 6:52:49 AM GMT-04:00
UnknownThursday, October 5th, 2023 at 6:52:49 AM GMT-04:00
GitHubThursday, October 5th, 2023 at 6:55:04 AM GMT-04:00
GitHubThursday, October 5th, 2023 at 7:36:56 AM GMT-04:00
UnknownThursday, October 5th, 2023 at 5:46:56 PM GMT-04:00
GitHubThursday, October 5th, 2023 at 10:54:09 AM GMT-04:00
UnknownSunday, October 8th, 2023 at 6:44:55 PM GMT-04:00
UnknownThursday, October 5th, 2023 at 4:13:10 PM GMT-04:00
GitHubThursday, October 5th, 2023 at 4:13:10 PM GMT-04:00
GitHubThursday, October 5th, 2023 at 4:15:14 PM GMT-04:00
GitHubThursday, October 5th, 2023 at 4:17:45 PM GMT-04:00
GitHubThursday, October 5th, 2023 at 5:27:43 PM GMT-04:00
GitHubThursday, October 5th, 2023 at 5:46:56 PM GMT-04:00
UnknownThursday, October 5th, 2023 at 5:46:56 PM GMT-04:00
GitHubThursday, October 5th, 2023 at 5:49:15 PM GMT-04:00
GitHubThursday, October 5th, 2023 at 5:50:31 PM GMT-04:00
GitHubThursday, October 5th, 2023 at 7:06:12 PM GMT-04:00
GitHubFriday, October 6th, 2023 at 4:29:35 AM GMT-04:00
GitHubFriday, October 6th, 2023 at 7:33:42 AM GMT-04:00
UnknownFriday, October 6th, 2023 at 7:33:42 AM GMT-04:00
UnknownFriday, October 6th, 2023 at 7:33:42 AM GMT-04:00
GitHubFriday, October 6th, 2023 at 7:35:43 AM GMT-04:00
GitHubFriday, October 6th, 2023 at 7:38:30 AM GMT-04:00
GitHubFriday, October 6th, 2023 at 8:26:48 AM GMT-04:00
UnknownMonday, October 9th, 2023 at 4:35:46 AM GMT-04:00
GitHubFriday, October 6th, 2023 at 9:18:00 AM GMT-04:00
UnknownFriday, October 20th, 2023 at 7:05:52 AM GMT-04:00
GitHubFriday, October 6th, 2023 at 10:00:40 AM GMT-04:00
UnknownTuesday, October 10th, 2023 at 1:26:33 PM GMT-04:00
GitHubSunday, October 8th, 2023 at 12:19:12 AM GMT-04:00
UnknownMonday, October 9th, 2023 at 2:23:18 AM GMT-04:00
GitHubSunday, October 8th, 2023 at 12:19:31 AM GMT-04:00
UnknownMonday, October 9th, 2023 at 2:23:47 AM GMT-04:00
GitHubSunday, October 8th, 2023 at 12:19:37 AM GMT-04:00
UnknownMonday, October 9th, 2023 at 2:23:37 AM GMT-04:00
GitHubSunday, October 8th, 2023 at 12:19:39 AM GMT-04:00
UnknownMonday, October 9th, 2023 at 3:28:20 AM GMT-04:00
GitHubSunday, October 8th, 2023 at 12:39:35 AM GMT-04:00
UnknownMonday, October 9th, 2023 at 2:20:49 AM GMT-04:00
GitHubSunday, October 8th, 2023 at 12:40:36 AM GMT-04:00
UnknownMonday, October 9th, 2023 at 4:36:40 AM GMT-04:00
GitHubSunday, October 8th, 2023 at 12:41:33 AM GMT-04:00
UnknownMonday, October 9th, 2023 at 2:21:15 AM GMT-04:00
UnknownSunday, October 8th, 2023 at 6:44:55 PM GMT-04:00
GitHubSunday, October 8th, 2023 at 6:44:55 PM GMT-04:00
GitHubSunday, October 8th, 2023 at 6:47:43 PM GMT-04:00
GitHubMonday, October 9th, 2023 at 12:16:01 AM GMT-04:00
UnknownThursday, October 19th, 2023 at 2:20:45 AM GMT-04:00
UnknownMonday, October 9th, 2023 at 2:20:49 AM GMT-04:00
GitHubMonday, October 9th, 2023 at 2:20:50 AM GMT-04:00
UnknownMonday, October 9th, 2023 at 2:21:15 AM GMT-04:00
GitHubMonday, October 9th, 2023 at 2:21:16 AM GMT-04:00
GitHubMonday, October 9th, 2023 at 2:22:48 AM GMT-04:00
GitHubMonday, October 9th, 2023 at 2:23:13 AM GMT-04:00
UnknownMonday, October 9th, 2023 at 2:23:18 AM GMT-04:00
UnknownMonday, October 9th, 2023 at 2:23:37 AM GMT-04:00
UnknownMonday, October 9th, 2023 at 2:23:47 AM GMT-04:00
GitHubMonday, October 9th, 2023 at 2:25:30 AM GMT-04:00
GitHubMonday, October 9th, 2023 at 2:25:37 AM GMT-04:00
UnknownMonday, October 9th, 2023 at 3:28:20 AM GMT-04:00
GitHubMonday, October 9th, 2023 at 3:28:21 AM GMT-04:00
GitHubMonday, October 9th, 2023 at 3:30:02 AM GMT-04:00
UnknownMonday, October 9th, 2023 at 3:31:35 AM GMT-04:00
GitHubMonday, October 9th, 2023 at 3:31:36 AM GMT-04:00
GitHubMonday, October 9th, 2023 at 3:32:26 AM GMT-04:00
GitHubMonday, October 9th, 2023 at 3:33:57 AM GMT-04:00
UnknownMonday, October 9th, 2023 at 4:35:46 AM GMT-04:00
GitHubMonday, October 9th, 2023 at 4:35:46 AM GMT-04:00
UnknownMonday, October 9th, 2023 at 4:35:47 AM GMT-04:00
UnknownMonday, October 9th, 2023 at 4:35:47 AM GMT-04:00
GitHubMonday, October 9th, 2023 at 4:35:47 AM GMT-04:00
UnknownMonday, October 9th, 2023 at 4:36:40 AM GMT-04:00
GitHubMonday, October 9th, 2023 at 4:36:40 AM GMT-04:00
GitHubMonday, October 9th, 2023 at 4:37:37 AM GMT-04:00
GitHubMonday, October 9th, 2023 at 4:38:38 AM GMT-04:00
GitHubMonday, October 9th, 2023 at 4:40:13 AM GMT-04:00
GitHubMonday, October 9th, 2023 at 4:40:56 AM GMT-04:00
GitHubMonday, October 9th, 2023 at 7:54:56 AM GMT-04:00
UnknownThursday, October 12th, 2023 at 4:44:41 AM GMT-04:00
GitHubMonday, October 9th, 2023 at 9:47:28 AM GMT-04:00
UnknownMonday, October 9th, 2023 at 9:58:11 AM GMT-04:00
UnknownMonday, October 9th, 2023 at 9:58:11 AM GMT-04:00
GitHubMonday, October 9th, 2023 at 9:58:12 AM GMT-04:00
GitHubMonday, October 9th, 2023 at 9:59:11 AM GMT-04:00
GitHubMonday, October 9th, 2023 at 10:00:28 AM GMT-04:00
GitHubMonday, October 9th, 2023 at 10:01:15 AM GMT-04:00
GitHubMonday, October 9th, 2023 at 10:03:33 AM GMT-04:00
GitHubMonday, October 9th, 2023 at 1:33:03 PM GMT-04:00
UnknownMonday, October 9th, 2023 at 2:32:45 PM GMT-04:00
GitHubMonday, October 9th, 2023 at 2:32:45 PM GMT-04:00
UnknownMonday, October 9th, 2023 at 2:32:45 PM GMT-04:00
GitHubMonday, October 9th, 2023 at 2:34:45 PM GMT-04:00
GitHubMonday, October 9th, 2023 at 2:37:17 PM GMT-04:00
GitHubMonday, October 9th, 2023 at 2:37:37 PM GMT-04:00
UnknownMonday, October 9th, 2023 at 2:42:15 PM GMT-04:00
UnknownMonday, October 9th, 2023 at 2:42:15 PM GMT-04:00
GitHubMonday, October 9th, 2023 at 2:42:15 PM GMT-04:00
GitHubMonday, October 9th, 2023 at 2:43:19 PM GMT-04:00
GitHubMonday, October 9th, 2023 at 2:45:38 PM GMT-04:00
GitHubMonday, October 9th, 2023 at 2:47:25 PM GMT-04:00
GitHubMonday, October 9th, 2023 at 2:47:56 PM GMT-04:00
GitHubMonday, October 9th, 2023 at 2:49:57 PM GMT-04:00
GitHubMonday, October 9th, 2023 at 3:16:16 PM GMT-04:00
GitHubTuesday, October 10th, 2023 at 7:55:33 AM GMT-04:00
UnknownTuesday, October 10th, 2023 at 7:55:33 AM GMT-04:00
UnknownWednesday, October 11th, 2023 at 2:04:28 AM GMT-04:00
UnknownTuesday, October 10th, 2023 at 7:55:33 AM GMT-04:00
UnknownTuesday, October 10th, 2023 at 1:26:33 PM GMT-04:00
GitHubTuesday, October 10th, 2023 at 1:26:33 PM GMT-04:00
GitHubTuesday, October 10th, 2023 at 1:28:56 PM GMT-04:00
GitHubTuesday, October 10th, 2023 at 2:43:56 PM GMT-04:00
UnknownWednesday, October 11th, 2023 at 1:24:47 PM GMT-04:00
UnknownWednesday, October 11th, 2023 at 2:04:28 AM GMT-04:00
GitHubWednesday, October 11th, 2023 at 2:04:28 AM GMT-04:00
GitHubWednesday, October 11th, 2023 at 2:06:33 AM GMT-04:00
GitHubWednesday, October 11th, 2023 at 2:08:54 AM GMT-04:00
GitHubWednesday, October 11th, 2023 at 2:17:24 AM GMT-04:00
UnknownThursday, November 9th, 2023 at 10:15:37 AM GMT-05:00
GitHubWednesday, October 11th, 2023 at 6:34:27 AM GMT-04:00
UnknownWednesday, October 11th, 2023 at 6:34:27 AM GMT-04:00
UnknownMonday, October 16th, 2023 at 2:48:04 AM GMT-04:00
UnknownWednesday, October 11th, 2023 at 6:34:27 AM GMT-04:00
GitHubWednesday, October 11th, 2023 at 1:24:47 PM GMT-04:00
UnknownWednesday, October 11th, 2023 at 1:24:47 PM GMT-04:00
GitHubWednesday, October 11th, 2023 at 1:30:07 PM GMT-04:00
UnknownThursday, October 12th, 2023 at 4:44:41 AM GMT-04:00
GitHubThursday, October 12th, 2023 at 4:44:41 AM GMT-04:00
GitHubThursday, October 12th, 2023 at 4:46:38 AM GMT-04:00
GitHubThursday, October 12th, 2023 at 4:49:14 AM GMT-04:00
GitHubThursday, October 12th, 2023 at 10:56:54 AM GMT-04:00
UnknownThursday, October 19th, 2023 at 3:33:04 PM GMT-04:00
GitHubFriday, October 13th, 2023 at 8:53:44 AM GMT-04:00
UnknownFriday, October 13th, 2023 at 9:36:57 AM GMT-04:00
GitHubFriday, October 13th, 2023 at 9:14:54 AM GMT-04:00
GitHubFriday, October 13th, 2023 at 9:18:52 AM GMT-04:00
UnknownFriday, October 13th, 2023 at 9:18:53 AM GMT-04:00
UnknownFriday, October 13th, 2023 at 9:18:53 AM GMT-04:00
GitHubFriday, October 13th, 2023 at 9:28:45 AM GMT-04:00
UnknownFriday, October 13th, 2023 at 9:36:57 AM GMT-04:00
GitHubFriday, October 13th, 2023 at 9:36:59 AM GMT-04:00
GitHubFriday, October 13th, 2023 at 9:39:42 AM GMT-04:00
GitHubFriday, October 13th, 2023 at 1:03:14 PM GMT-04:00
GitHubSunday, October 15th, 2023 at 12:05:12 AM GMT-04:00
UnknownMonday, October 16th, 2023 at 2:04:25 AM GMT-04:00
GitHubSunday, October 15th, 2023 at 12:05:13 AM GMT-04:00
UnknownMonday, October 16th, 2023 at 2:12:15 AM GMT-04:00
GitHubSunday, October 15th, 2023 at 12:18:47 AM GMT-04:00
UnknownWednesday, October 18th, 2023 at 2:05:19 AM GMT-04:00
GitHubSunday, October 15th, 2023 at 12:42:08 AM GMT-04:00
UnknownMonday, October 16th, 2023 at 2:04:56 AM GMT-04:00
GitHubSunday, October 15th, 2023 at 12:42:16 AM GMT-04:00
UnknownMonday, October 16th, 2023 at 2:05:55 AM GMT-04:00
GitHubSunday, October 15th, 2023 at 12:42:28 AM GMT-04:00
UnknownMonday, October 16th, 2023 at 2:05:25 AM GMT-04:00
GitHubSunday, October 15th, 2023 at 12:52:00 AM GMT-04:00
UnknownMonday, October 16th, 2023 at 2:02:31 AM GMT-04:00
GitHubSunday, October 15th, 2023 at 12:52:01 AM GMT-04:00
UnknownMonday, October 16th, 2023 at 2:06:19 AM GMT-04:00
GitHubMonday, October 16th, 2023 at 12:44:43 AM GMT-04:00
GitHubMonday, October 16th, 2023 at 12:44:54 AM GMT-04:00
UnknownMonday, October 16th, 2023 at 2:02:31 AM GMT-04:00
GitHubMonday, October 16th, 2023 at 2:02:32 AM GMT-04:00
UnknownMonday, October 16th, 2023 at 2:04:25 AM GMT-04:00
GitHubMonday, October 16th, 2023 at 2:04:29 AM GMT-04:00
GitHubMonday, October 16th, 2023 at 2:04:55 AM GMT-04:00
UnknownMonday, October 16th, 2023 at 2:04:56 AM GMT-04:00
GitHubMonday, October 16th, 2023 at 2:05:25 AM GMT-04:00
UnknownMonday, October 16th, 2023 at 2:05:25 AM GMT-04:00
UnknownMonday, October 16th, 2023 at 2:05:55 AM GMT-04:00
UnknownMonday, October 16th, 2023 at 2:06:19 AM GMT-04:00
GitHubMonday, October 16th, 2023 at 2:06:57 AM GMT-04:00
GitHubMonday, October 16th, 2023 at 2:07:11 AM GMT-04:00
GitHubMonday, October 16th, 2023 at 2:07:17 AM GMT-04:00
GitHubMonday, October 16th, 2023 at 2:09:18 AM GMT-04:00
GitHubMonday, October 16th, 2023 at 2:10:01 AM GMT-04:00
UnknownMonday, October 16th, 2023 at 2:12:15 AM GMT-04:00
UnknownMonday, October 16th, 2023 at 2:48:04 AM GMT-04:00
UnknownMonday, October 16th, 2023 at 4:30:59 AM GMT-04:00
GitHubMonday, October 16th, 2023 at 4:30:59 AM GMT-04:00
UnknownMonday, October 16th, 2023 at 4:30:59 AM GMT-04:00
UnknownMonday, October 16th, 2023 at 4:30:59 AM GMT-04:00
GitHubMonday, October 16th, 2023 at 4:31:00 AM GMT-04:00
GitHubMonday, October 16th, 2023 at 4:32:49 AM GMT-04:00
GitHubMonday, October 16th, 2023 at 4:35:07 AM GMT-04:00
GitHubMonday, October 16th, 2023 at 8:02:57 AM GMT-04:00
UnknownMonday, October 16th, 2023 at 8:02:57 AM GMT-04:00
UnknownMonday, October 23rd, 2023 at 5:57:30 AM GMT-04:00
UnknownMonday, October 16th, 2023 at 8:02:57 AM GMT-04:00
GitHubMonday, October 16th, 2023 at 10:19:13 AM GMT-04:00
UnknownMonday, October 16th, 2023 at 4:59:56 PM GMT-04:00
UnknownMonday, October 16th, 2023 at 4:59:56 PM GMT-04:00
GitHubMonday, October 16th, 2023 at 4:59:56 PM GMT-04:00
GitHubMonday, October 16th, 2023 at 5:02:41 PM GMT-04:00
GitHubTuesday, October 17th, 2023 at 4:35:32 AM GMT-04:00
UnknownTuesday, October 17th, 2023 at 4:35:33 AM GMT-04:00
UnknownWednesday, October 25th, 2023 at 3:34:22 AM GMT-04:00
UnknownTuesday, October 17th, 2023 at 4:35:33 AM GMT-04:00
UnknownWednesday, October 18th, 2023 at 2:05:19 AM GMT-04:00
GitHubWednesday, October 18th, 2023 at 10:29:38 AM GMT-04:00
GitHubThursday, October 19th, 2023 at 2:20:44 AM GMT-04:00
UnknownThursday, October 19th, 2023 at 2:20:45 AM GMT-04:00
GitHubThursday, October 19th, 2023 at 2:22:57 AM GMT-04:00
GitHubThursday, October 19th, 2023 at 2:25:34 AM GMT-04:00
GitHubThursday, October 19th, 2023 at 5:43:58 AM GMT-04:00
UnknownThursday, October 19th, 2023 at 5:43:58 AM GMT-04:00
UnknownThursday, October 26th, 2023 at 5:48:16 AM GMT-04:00
UnknownThursday, October 19th, 2023 at 5:43:58 AM GMT-04:00
GitHubThursday, October 19th, 2023 at 8:01:50 AM GMT-04:00
UnknownThursday, October 19th, 2023 at 8:01:51 AM GMT-04:00
UnknownFriday, October 20th, 2023 at 7:05:52 AM GMT-04:00
UnknownThursday, October 19th, 2023 at 8:01:51 AM GMT-04:00
GitHubThursday, October 19th, 2023 at 3:33:03 PM GMT-04:00
UnknownThursday, October 19th, 2023 at 3:33:04 PM GMT-04:00
GitHubThursday, October 19th, 2023 at 3:35:39 PM GMT-04:00
GitHubThursday, October 19th, 2023 at 3:49:03 PM GMT-04:00
UnknownThursday, October 19th, 2023 at 4:25:12 PM GMT-04:00
GitHubThursday, October 19th, 2023 at 4:25:11 PM GMT-04:00
UnknownThursday, October 19th, 2023 at 4:25:12 PM GMT-04:00
GitHubThursday, October 19th, 2023 at 4:27:24 PM GMT-04:00
GitHubThursday, October 19th, 2023 at 5:52:21 PM GMT-04:00
UnknownThursday, October 19th, 2023 at 5:55:05 PM GMT-04:00
UnknownThursday, October 19th, 2023 at 5:55:05 PM GMT-04:00
GitHubThursday, October 19th, 2023 at 5:55:05 PM GMT-04:00
GitHubThursday, October 19th, 2023 at 5:57:44 PM GMT-04:00
GitHubFriday, October 20th, 2023 at 6:58:41 AM GMT-04:00
UnknownMonday, October 23rd, 2023 at 3:45:25 AM GMT-04:00
GitHubFriday, October 20th, 2023 at 7:05:52 AM GMT-04:00
UnknownFriday, October 20th, 2023 at 7:05:52 AM GMT-04:00
UnknownFriday, October 20th, 2023 at 7:05:52 AM GMT-04:00
GitHubFriday, October 20th, 2023 at 7:07:57 AM GMT-04:00
GitHubFriday, October 20th, 2023 at 7:10:17 AM GMT-04:00
GitHubFriday, October 20th, 2023 at 7:49:50 AM GMT-04:00
UnknownMonday, October 23rd, 2023 at 3:46:24 AM GMT-04:00
GitHubFriday, October 20th, 2023 at 10:03:38 AM GMT-04:00
GitHubSunday, October 22nd, 2023 at 12:01:45 AM GMT-04:00
UnknownMonday, October 23rd, 2023 at 3:51:04 AM GMT-04:00
GitHubSunday, October 22nd, 2023 at 12:01:53 AM GMT-04:00
UnknownMonday, October 23rd, 2023 at 3:51:51 AM GMT-04:00
GitHubSunday, October 22nd, 2023 at 12:35:40 AM GMT-04:00
UnknownMonday, October 23rd, 2023 at 3:52:39 AM GMT-04:00
UnknownMonday, October 23rd, 2023 at 3:45:25 AM GMT-04:00
GitHubMonday, October 23rd, 2023 at 3:46:23 AM GMT-04:00
UnknownMonday, October 23rd, 2023 at 3:46:24 AM GMT-04:00
GitHubMonday, October 23rd, 2023 at 3:48:32 AM GMT-04:00
GitHubMonday, October 23rd, 2023 at 3:50:45 AM GMT-04:00
GitHubMonday, October 23rd, 2023 at 3:51:03 AM GMT-04:00
UnknownMonday, October 23rd, 2023 at 3:51:04 AM GMT-04:00
GitHubMonday, October 23rd, 2023 at 3:51:50 AM GMT-04:00
UnknownMonday, October 23rd, 2023 at 3:51:51 AM GMT-04:00
UnknownMonday, October 23rd, 2023 at 3:52:39 AM GMT-04:00
GitHubMonday, October 23rd, 2023 at 3:52:40 AM GMT-04:00
GitHubMonday, October 23rd, 2023 at 3:52:48 AM GMT-04:00
GitHubMonday, October 23rd, 2023 at 3:53:57 AM GMT-04:00
GitHubMonday, October 23rd, 2023 at 3:54:55 AM GMT-04:00
GitHubMonday, October 23rd, 2023 at 3:56:33 AM GMT-04:00
GitHubMonday, October 23rd, 2023 at 4:00:17 AM GMT-04:00
GitHubMonday, October 23rd, 2023 at 4:02:53 AM GMT-04:00
GitHubMonday, October 23rd, 2023 at 5:57:30 AM GMT-04:00
UnknownMonday, October 23rd, 2023 at 5:57:30 AM GMT-04:00
GitHubMonday, October 23rd, 2023 at 12:55:33 PM GMT-04:00
UnknownMonday, October 23rd, 2023 at 5:01:17 PM GMT-04:00
GitHubMonday, October 23rd, 2023 at 5:01:15 PM GMT-04:00
UnknownMonday, October 23rd, 2023 at 5:01:17 PM GMT-04:00
GitHubMonday, October 23rd, 2023 at 5:03:52 PM GMT-04:00
GitHubTuesday, October 24th, 2023 at 9:45:42 AM GMT-04:00
UnknownMonday, October 30th, 2023 at 5:05:57 PM GMT-04:00
GitHubTuesday, October 24th, 2023 at 10:35:38 AM GMT-04:00
UnknownMonday, October 30th, 2023 at 5:05:35 PM GMT-04:00
GitHubTuesday, October 24th, 2023 at 7:34:26 PM GMT-04:00
UnknownThursday, October 26th, 2023 at 5:49:47 AM GMT-04:00
GitHubTuesday, October 24th, 2023 at 11:57:46 PM GMT-04:00
UnknownFriday, November 10th, 2023 at 7:19:42 AM GMT-05:00
GitHubWednesday, October 25th, 2023 at 3:34:21 AM GMT-04:00
UnknownWednesday, October 25th, 2023 at 3:34:22 AM GMT-04:00
UnknownWednesday, October 25th, 2023 at 3:34:22 AM GMT-04:00
GitHubWednesday, October 25th, 2023 at 3:36:08 AM GMT-04:00
GitHubWednesday, October 25th, 2023 at 3:38:41 AM GMT-04:00
GitHubWednesday, October 25th, 2023 at 3:40:40 AM GMT-04:00
UnknownThursday, October 26th, 2023 at 5:40:57 AM GMT-04:00
GitHubWednesday, October 25th, 2023 at 5:56:06 PM GMT-04:00
GitHubThursday, October 26th, 2023 at 4:25:10 AM GMT-04:00
UnknownThursday, October 26th, 2023 at 4:25:10 AM GMT-04:00
UnknownThursday, October 26th, 2023 at 5:40:57 AM GMT-04:00
UnknownThursday, October 26th, 2023 at 4:25:10 AM GMT-04:00
GitHubThursday, October 26th, 2023 at 5:40:56 AM GMT-04:00
UnknownThursday, October 26th, 2023 at 5:40:57 AM GMT-04:00
UnknownThursday, October 26th, 2023 at 5:40:57 AM GMT-04:00
GitHubThursday, October 26th, 2023 at 5:42:42 AM GMT-04:00
GitHubThursday, October 26th, 2023 at 5:45:05 AM GMT-04:00
GitHubThursday, October 26th, 2023 at 5:48:16 AM GMT-04:00
UnknownThursday, October 26th, 2023 at 5:48:16 AM GMT-04:00
UnknownThursday, October 26th, 2023 at 5:49:47 AM GMT-04:00
GitHubThursday, October 26th, 2023 at 5:50:50 AM GMT-04:00
GitHubThursday, October 26th, 2023 at 5:53:55 AM GMT-04:00
GitHubThursday, October 26th, 2023 at 6:16:53 AM GMT-04:00
UnknownThursday, October 26th, 2023 at 6:35:14 AM GMT-04:00
UnknownThursday, October 26th, 2023 at 7:48:49 AM GMT-04:00
UnknownThursday, October 26th, 2023 at 6:35:14 AM GMT-04:00
GitHubThursday, October 26th, 2023 at 7:08:27 AM GMT-04:00
GitHubThursday, October 26th, 2023 at 7:08:27 AM GMT-04:00
UnknownThursday, October 26th, 2023 at 7:08:27 AM GMT-04:00
UnknownThursday, October 26th, 2023 at 7:08:27 AM GMT-04:00
GitHubThursday, October 26th, 2023 at 7:10:52 AM GMT-04:00
GitHubThursday, October 26th, 2023 at 7:13:36 AM GMT-04:00
UnknownThursday, October 26th, 2023 at 7:48:49 AM GMT-04:00
GitHubFriday, October 27th, 2023 at 5:00:21 AM GMT-04:00
UnknownWednesday, November 15th, 2023 at 4:58:47 AM GMT-05:00
GitHubFriday, October 27th, 2023 at 9:23:56 AM GMT-04:00
UnknownFriday, October 27th, 2023 at 9:23:56 AM GMT-04:00
UnknownFriday, October 27th, 2023 at 9:33:24 AM GMT-04:00
UnknownFriday, October 27th, 2023 at 9:23:56 AM GMT-04:00
GitHubFriday, October 27th, 2023 at 9:33:23 AM GMT-04:00
UnknownFriday, October 27th, 2023 at 9:33:24 AM GMT-04:00
GitHubFriday, October 27th, 2023 at 9:36:02 AM GMT-04:00
GitHubFriday, October 27th, 2023 at 9:38:18 AM GMT-04:00
GitHubFriday, October 27th, 2023 at 12:55:58 PM GMT-04:00
UnknownMonday, October 30th, 2023 at 9:05:29 AM GMT-04:00
GitHubSunday, October 29th, 2023 at 12:24:29 AM GMT-04:00
UnknownMonday, October 30th, 2023 at 2:53:06 AM GMT-04:00
GitHubSunday, October 29th, 2023 at 11:15:19 PM GMT-04:00
UnknownTuesday, October 31st, 2023 at 2:32:45 AM GMT-04:00
GitHubMonday, October 30th, 2023 at 12:45:40 AM GMT-04:00
GitHubMonday, October 30th, 2023 at 2:53:05 AM GMT-04:00
UnknownMonday, October 30th, 2023 at 2:53:06 AM GMT-04:00
GitHubMonday, October 30th, 2023 at 2:55:39 AM GMT-04:00
GitHubMonday, October 30th, 2023 at 2:58:20 AM GMT-04:00
GitHubMonday, October 30th, 2023 at 5:23:38 AM GMT-04:00
UnknownTuesday, October 31st, 2023 at 2:34:07 AM GMT-04:00
GitHubMonday, October 30th, 2023 at 9:05:29 AM GMT-04:00
UnknownMonday, October 30th, 2023 at 9:05:29 AM GMT-04:00
GitHubMonday, October 30th, 2023 at 9:07:50 AM GMT-04:00
GitHubMonday, October 30th, 2023 at 11:17:21 AM GMT-04:00
UnknownMonday, October 30th, 2023 at 5:17:18 PM GMT-04:00
GitHubMonday, October 30th, 2023 at 11:36:04 AM GMT-04:00
UnknownFriday, November 10th, 2023 at 4:35:54 AM GMT-05:00
GitHubMonday, October 30th, 2023 at 12:21:47 PM GMT-04:00
UnknownMonday, October 30th, 2023 at 5:05:16 PM GMT-04:00
GitHubMonday, October 30th, 2023 at 2:31:44 PM GMT-04:00
UnknownThursday, November 2nd, 2023 at 6:26:04 AM GMT-04:00
GitHubMonday, October 30th, 2023 at 5:05:16 PM GMT-04:00
UnknownMonday, October 30th, 2023 at 5:05:16 PM GMT-04:00
GitHubMonday, October 30th, 2023 at 5:05:35 PM GMT-04:00
UnknownMonday, October 30th, 2023 at 5:05:35 PM GMT-04:00
GitHubMonday, October 30th, 2023 at 5:05:57 PM GMT-04:00
UnknownMonday, October 30th, 2023 at 5:05:57 PM GMT-04:00
GitHubMonday, October 30th, 2023 at 5:08:13 PM GMT-04:00
GitHubMonday, October 30th, 2023 at 5:17:17 PM GMT-04:00
UnknownMonday, October 30th, 2023 at 5:17:18 PM GMT-04:00
GitHubMonday, October 30th, 2023 at 5:20:26 PM GMT-04:00
GitHubMonday, October 30th, 2023 at 5:23:11 PM GMT-04:00
GitHubMonday, October 30th, 2023 at 6:49:10 PM GMT-04:00
UnknownTuesday, October 31st, 2023 at 8:36:45 AM GMT-04:00
GitHubTuesday, October 31st, 2023 at 2:32:45 AM GMT-04:00
UnknownTuesday, October 31st, 2023 at 2:32:45 AM GMT-04:00
GitHubTuesday, October 31st, 2023 at 2:34:07 AM GMT-04:00
UnknownTuesday, October 31st, 2023 at 2:34:07 AM GMT-04:00
GitHubTuesday, October 31st, 2023 at 2:35:31 AM GMT-04:00
GitHubTuesday, October 31st, 2023 at 2:36:44 AM GMT-04:00
GitHubTuesday, October 31st, 2023 at 8:36:45 AM GMT-04:00
UnknownTuesday, October 31st, 2023 at 8:36:45 AM GMT-04:00
GitHubTuesday, October 31st, 2023 at 8:39:28 AM GMT-04:00
GitHubTuesday, October 31st, 2023 at 8:42:00 AM GMT-04:00
GitHubTuesday, October 31st, 2023 at 8:06:37 PM GMT-04:00
UnknownWednesday, November 15th, 2023 at 4:56:42 AM GMT-05:00
GitHubWednesday, November 1st, 2023 at 10:06:55 AM GMT-04:00
UnknownWednesday, November 8th, 2023 at 10:16:34 AM GMT-05:00
GitHubWednesday, November 1st, 2023 at 12:28:09 PM GMT-04:00
UnknownThursday, November 2nd, 2023 at 5:42:35 AM GMT-04:00
GitHubThursday, November 2nd, 2023 at 5:42:34 AM GMT-04:00
UnknownThursday, November 2nd, 2023 at 5:42:35 AM GMT-04:00
GitHubThursday, November 2nd, 2023 at 5:45:57 AM GMT-04:00
GitHubThursday, November 2nd, 2023 at 5:46:19 AM GMT-04:00
GitHubThursday, November 2nd, 2023 at 5:48:27 AM GMT-04:00
GitHubThursday, November 2nd, 2023 at 5:48:58 AM GMT-04:00
GitHubThursday, November 2nd, 2023 at 5:51:51 AM GMT-04:00
GitHubThursday, November 2nd, 2023 at 6:10:25 AM GMT-04:00
UnknownThursday, November 2nd, 2023 at 6:26:04 AM GMT-04:00
GitHubThursday, November 2nd, 2023 at 6:26:04 AM GMT-04:00
GitHubThursday, November 2nd, 2023 at 6:28:34 AM GMT-04:00
GitHubThursday, November 2nd, 2023 at 6:38:03 AM GMT-04:00
UnknownThursday, November 2nd, 2023 at 6:50:29 AM GMT-04:00
GitHubThursday, November 2nd, 2023 at 6:50:27 AM GMT-04:00
UnknownThursday, November 2nd, 2023 at 6:50:29 AM GMT-04:00
GitHubThursday, November 2nd, 2023 at 6:53:15 AM GMT-04:00
GitHubFriday, November 3rd, 2023 at 7:31:33 AM GMT-04:00
GitHubFriday, November 3rd, 2023 at 7:43:08 AM GMT-04:00
UnknownThursday, November 9th, 2023 at 5:03:09 AM GMT-05:00
GitHubFriday, November 3rd, 2023 at 10:38:19 AM GMT-04:00
UnknownFriday, November 3rd, 2023 at 2:14:40 PM GMT-04:00
GitHubFriday, November 3rd, 2023 at 1:26:17 PM GMT-04:00
GitHubFriday, November 3rd, 2023 at 1:32:04 PM GMT-04:00
GitHubFriday, November 3rd, 2023 at 1:32:59 PM GMT-04:00
GitHubFriday, November 3rd, 2023 at 1:34:40 PM GMT-04:00
UnknownFriday, November 3rd, 2023 at 1:34:40 PM GMT-04:00
UnknownFriday, November 3rd, 2023 at 1:34:40 PM GMT-04:00
UnknownFriday, November 3rd, 2023 at 2:14:40 PM GMT-04:00
GitHubFriday, November 3rd, 2023 at 2:40:40 PM GMT-04:00
UnknownFriday, November 3rd, 2023 at 2:43:56 PM GMT-04:00
GitHubFriday, November 3rd, 2023 at 2:43:10 PM GMT-04:00
UnknownFriday, November 3rd, 2023 at 2:43:56 PM GMT-04:00
GitHubFriday, November 3rd, 2023 at 2:50:51 PM GMT-04:00
GitHubFriday, November 3rd, 2023 at 2:50:54 PM GMT-04:00
UnknownFriday, November 3rd, 2023 at 2:50:55 PM GMT-04:00
UnknownFriday, November 3rd, 2023 at 2:50:55 PM GMT-04:00
GitHubFriday, November 3rd, 2023 at 4:55:02 PM GMT-04:00
GitHubFriday, November 3rd, 2023 at 6:30:02 PM GMT-04:00
UnknownWednesday, November 8th, 2023 at 7:56:49 AM GMT-05:00
GitHubFriday, November 3rd, 2023 at 6:34:38 PM GMT-04:00
UnknownWednesday, November 8th, 2023 at 7:56:50 AM GMT-05:00
GitHubSunday, November 5th, 2023 at 12:03:50 AM GMT-04:00
UnknownTuesday, November 7th, 2023 at 7:16:58 AM GMT-05:00
GitHubMonday, November 6th, 2023 at 9:20:37 AM GMT-05:00
UnknownMonday, November 13th, 2023 at 4:15:17 AM GMT-05:00
GitHubTuesday, November 7th, 2023 at 7:16:57 AM GMT-05:00
UnknownTuesday, November 7th, 2023 at 7:16:58 AM GMT-05:00
GitHubTuesday, November 7th, 2023 at 7:19:18 AM GMT-05:00
GitHubTuesday, November 7th, 2023 at 7:21:16 AM GMT-05:00
GitHubTuesday, November 7th, 2023 at 10:08:25 AM GMT-05:00
UnknownTuesday, November 7th, 2023 at 10:08:26 AM GMT-05:00
UnknownWednesday, November 8th, 2023 at 1:29:17 AM GMT-05:00
UnknownTuesday, November 7th, 2023 at 10:08:26 AM GMT-05:00
GitHubWednesday, November 8th, 2023 at 1:29:17 AM GMT-05:00
UnknownWednesday, November 8th, 2023 at 1:29:17 AM GMT-05:00
GitHubWednesday, November 8th, 2023 at 2:56:00 AM GMT-05:00
UnknownWednesday, November 8th, 2023 at 2:56:00 AM GMT-05:00
UnknownWednesday, November 8th, 2023 at 6:15:34 AM GMT-05:00
UnknownWednesday, November 8th, 2023 at 2:56:00 AM GMT-05:00
GitHubWednesday, November 8th, 2023 at 3:42:06 AM GMT-05:00
UnknownWednesday, November 8th, 2023 at 4:05:30 AM GMT-05:00
UnknownWednesday, November 8th, 2023 at 4:05:30 AM GMT-05:00
GitHubWednesday, November 8th, 2023 at 4:05:30 AM GMT-05:00
GitHubWednesday, November 8th, 2023 at 4:08:09 AM GMT-05:00
GitHubWednesday, November 8th, 2023 at 4:10:00 AM GMT-05:00
GitHubWednesday, November 8th, 2023 at 6:15:33 AM GMT-05:00
UnknownWednesday, November 8th, 2023 at 6:15:34 AM GMT-05:00
GitHubWednesday, November 8th, 2023 at 6:18:00 AM GMT-05:00
GitHubWednesday, November 8th, 2023 at 6:20:22 AM GMT-05:00
GitHubWednesday, November 8th, 2023 at 7:56:48 AM GMT-05:00
UnknownWednesday, November 8th, 2023 at 7:56:49 AM GMT-05:00
UnknownWednesday, November 8th, 2023 at 7:56:50 AM GMT-05:00
GitHubWednesday, November 8th, 2023 at 7:59:57 AM GMT-05:00
GitHubWednesday, November 8th, 2023 at 8:02:29 AM GMT-05:00
GitHubWednesday, November 8th, 2023 at 10:16:33 AM GMT-05:00
UnknownWednesday, November 8th, 2023 at 10:16:34 AM GMT-05:00
GitHubWednesday, November 8th, 2023 at 4:32:47 PM GMT-05:00
UnknownThursday, November 9th, 2023 at 5:08:05 AM GMT-05:00
GitHubThursday, November 9th, 2023 at 5:03:08 AM GMT-05:00
UnknownThursday, November 9th, 2023 at 5:03:09 AM GMT-05:00
GitHubThursday, November 9th, 2023 at 5:08:05 AM GMT-05:00
UnknownThursday, November 9th, 2023 at 5:08:05 AM GMT-05:00
UnknownThursday, November 9th, 2023 at 10:15:37 AM GMT-05:00
GitHubThursday, November 9th, 2023 at 11:33:22 AM GMT-05:00
UnknownMonday, November 13th, 2023 at 2:32:08 PM GMT-05:00
GitHubThursday, November 9th, 2023 at 3:29:23 PM GMT-05:00
GitHubThursday, November 9th, 2023 at 4:03:16 PM GMT-05:00
UnknownFriday, November 10th, 2023 at 4:23:06 AM GMT-05:00
GitHubFriday, November 10th, 2023 at 4:23:05 AM GMT-05:00
UnknownFriday, November 10th, 2023 at 4:23:06 AM GMT-05:00
GitHubFriday, November 10th, 2023 at 4:26:31 AM GMT-05:00
GitHubFriday, November 10th, 2023 at 4:28:26 AM GMT-05:00
GitHubFriday, November 10th, 2023 at 4:35:53 AM GMT-05:00
UnknownFriday, November 10th, 2023 at 4:35:54 AM GMT-05:00
GitHubFriday, November 10th, 2023 at 4:35:54 AM GMT-05:00
UnknownFriday, November 10th, 2023 at 4:35:54 AM GMT-05:00
UnknownFriday, November 10th, 2023 at 4:35:54 AM GMT-05:00
GitHubFriday, November 10th, 2023 at 4:38:11 AM GMT-05:00
GitHubFriday, November 10th, 2023 at 4:40:07 AM GMT-05:00
GitHubFriday, November 10th, 2023 at 4:47:06 AM GMT-05:00
UnknownFriday, November 10th, 2023 at 5:11:07 AM GMT-05:00
GitHubFriday, November 10th, 2023 at 5:11:07 AM GMT-05:00
UnknownFriday, November 10th, 2023 at 5:11:07 AM GMT-05:00
GitHubFriday, November 10th, 2023 at 5:13:45 AM GMT-05:00
GitHubFriday, November 10th, 2023 at 5:15:41 AM GMT-05:00
GitHubFriday, November 10th, 2023 at 7:19:41 AM GMT-05:00
UnknownFriday, November 10th, 2023 at 7:19:42 AM GMT-05:00
GitHubFriday, November 10th, 2023 at 7:20:07 AM GMT-05:00
UnknownMonday, November 13th, 2023 at 11:28:57 AM GMT-05:00
GitHubFriday, November 10th, 2023 at 7:22:27 AM GMT-05:00
GitHubFriday, November 10th, 2023 at 7:24:19 AM GMT-05:00
GitHubFriday, November 10th, 2023 at 10:11:13 AM GMT-05:00
UnknownFriday, November 10th, 2023 at 10:16:05 AM GMT-05:00
GitHubFriday, November 10th, 2023 at 10:16:05 AM GMT-05:00
UnknownFriday, November 10th, 2023 at 10:16:05 AM GMT-05:00
GitHubFriday, November 10th, 2023 at 4:01:19 PM GMT-05:00
GitHubSaturday, November 11th, 2023 at 11:28:57 PM GMT-05:00
UnknownMonday, November 13th, 2023 at 1:41:45 AM GMT-05:00
GitHubSaturday, November 11th, 2023 at 11:29:08 PM GMT-05:00
UnknownMonday, November 13th, 2023 at 1:41:28 AM GMT-05:00
GitHubSaturday, November 11th, 2023 at 11:29:33 PM GMT-05:00
UnknownMonday, November 13th, 2023 at 1:42:41 AM GMT-05:00
GitHubSaturday, November 11th, 2023 at 11:30:01 PM GMT-05:00
UnknownMonday, November 13th, 2023 at 5:26:08 AM GMT-05:00
GitHubSaturday, November 11th, 2023 at 11:30:02 PM GMT-05:00
UnknownMonday, November 13th, 2023 at 1:44:44 AM GMT-05:00
GitHubSaturday, November 11th, 2023 at 11:39:08 PM GMT-05:00
UnknownMonday, November 13th, 2023 at 1:45:27 AM GMT-05:00
GitHubSaturday, November 11th, 2023 at 11:39:22 PM GMT-05:00
UnknownMonday, November 13th, 2023 at 1:44:08 AM GMT-05:00
GitHubSaturday, November 11th, 2023 at 11:39:25 PM GMT-05:00
UnknownMonday, November 13th, 2023 at 1:40:43 AM GMT-05:00
GitHubSaturday, November 11th, 2023 at 11:39:42 PM GMT-05:00
UnknownMonday, November 13th, 2023 at 5:25:38 AM GMT-05:00
GitHubSaturday, November 11th, 2023 at 11:39:54 PM GMT-05:00
UnknownMonday, November 13th, 2023 at 11:32:29 AM GMT-05:00
GitHubMonday, November 13th, 2023 at 1:40:43 AM GMT-05:00
UnknownMonday, November 13th, 2023 at 1:40:43 AM GMT-05:00
GitHubMonday, November 13th, 2023 at 1:41:28 AM GMT-05:00
UnknownMonday, November 13th, 2023 at 1:41:28 AM GMT-05:00
GitHubMonday, November 13th, 2023 at 1:41:44 AM GMT-05:00
UnknownMonday, November 13th, 2023 at 1:41:45 AM GMT-05:00
GitHubMonday, November 13th, 2023 at 1:42:40 AM GMT-05:00
UnknownMonday, November 13th, 2023 at 1:42:41 AM GMT-05:00
GitHubMonday, November 13th, 2023 at 1:44:07 AM GMT-05:00
UnknownMonday, November 13th, 2023 at 1:44:08 AM GMT-05:00
GitHubMonday, November 13th, 2023 at 1:44:28 AM GMT-05:00
GitHubMonday, November 13th, 2023 at 1:44:38 AM GMT-05:00
GitHubMonday, November 13th, 2023 at 1:44:43 AM GMT-05:00
UnknownMonday, November 13th, 2023 at 1:44:44 AM GMT-05:00
GitHubMonday, November 13th, 2023 at 1:45:12 AM GMT-05:00
GitHubMonday, November 13th, 2023 at 1:45:27 AM GMT-05:00
UnknownMonday, November 13th, 2023 at 1:45:27 AM GMT-05:00
GitHubMonday, November 13th, 2023 at 1:46:13 AM GMT-05:00
GitHubMonday, November 13th, 2023 at 1:46:38 AM GMT-05:00
GitHubMonday, November 13th, 2023 at 1:47:00 AM GMT-05:00
GitHubMonday, November 13th, 2023 at 1:47:13 AM GMT-05:00
GitHubMonday, November 13th, 2023 at 1:48:50 AM GMT-05:00
GitHubMonday, November 13th, 2023 at 1:59:11 AM GMT-05:00
GitHubMonday, November 13th, 2023 at 2:01:33 AM GMT-05:00
GitHubMonday, November 13th, 2023 at 2:10:16 AM GMT-05:00
GitHubMonday, November 13th, 2023 at 2:12:58 AM GMT-05:00
GitHubMonday, November 13th, 2023 at 2:17:45 AM GMT-05:00
GitHubMonday, November 13th, 2023 at 2:20:40 AM GMT-05:00
UnknownMonday, November 13th, 2023 at 4:15:17 AM GMT-05:00
GitHubMonday, November 13th, 2023 at 5:25:38 AM GMT-05:00
UnknownMonday, November 13th, 2023 at 5:25:38 AM GMT-05:00
GitHubMonday, November 13th, 2023 at 5:26:08 AM GMT-05:00
UnknownMonday, November 13th, 2023 at 5:26:08 AM GMT-05:00
GitHubMonday, November 13th, 2023 at 5:29:27 AM GMT-05:00
GitHubMonday, November 13th, 2023 at 5:31:36 AM GMT-05:00
GitHubMonday, November 13th, 2023 at 5:52:02 AM GMT-05:00
UnknownMonday, November 13th, 2023 at 5:52:03 AM GMT-05:00
UnknownMonday, November 13th, 2023 at 5:52:03 AM GMT-05:00
GitHubMonday, November 13th, 2023 at 10:44:42 AM GMT-05:00
GitHubMonday, November 13th, 2023 at 11:28:56 AM GMT-05:00
UnknownMonday, November 13th, 2023 at 11:28:57 AM GMT-05:00
GitHubMonday, November 13th, 2023 at 11:29:21 AM GMT-05:00
UnknownMonday, November 13th, 2023 at 11:31:14 AM GMT-05:00
GitHubMonday, November 13th, 2023 at 11:29:22 AM GMT-05:00
UnknownMonday, November 13th, 2023 at 11:31:43 AM GMT-05:00
GitHubMonday, November 13th, 2023 at 11:29:27 AM GMT-05:00
UnknownMonday, November 13th, 2023 at 11:31:33 AM GMT-05:00
GitHubMonday, November 13th, 2023 at 11:30:11 AM GMT-05:00
UnknownMonday, November 13th, 2023 at 11:31:59 AM GMT-05:00
GitHubMonday, November 13th, 2023 at 11:31:14 AM GMT-05:00
UnknownMonday, November 13th, 2023 at 11:31:14 AM GMT-05:00
GitHubMonday, November 13th, 2023 at 11:31:33 AM GMT-05:00
UnknownMonday, November 13th, 2023 at 11:31:33 AM GMT-05:00
GitHubMonday, November 13th, 2023 at 11:31:43 AM GMT-05:00
UnknownMonday, November 13th, 2023 at 11:31:43 AM GMT-05:00
GitHubMonday, November 13th, 2023 at 11:31:51 AM GMT-05:00
GitHubMonday, November 13th, 2023 at 11:31:58 AM GMT-05:00
UnknownMonday, November 13th, 2023 at 11:31:59 AM GMT-05:00
GitHubMonday, November 13th, 2023 at 11:32:29 AM GMT-05:00
UnknownMonday, November 13th, 2023 at 11:32:29 AM GMT-05:00
GitHubMonday, November 13th, 2023 at 11:34:20 AM GMT-05:00
GitHubMonday, November 13th, 2023 at 11:34:45 AM GMT-05:00
GitHubMonday, November 13th, 2023 at 11:35:17 AM GMT-05:00
GitHubMonday, November 13th, 2023 at 11:36:35 AM GMT-05:00
GitHubMonday, November 13th, 2023 at 11:37:00 AM GMT-05:00
GitHubMonday, November 13th, 2023 at 11:37:12 AM GMT-05:00
GitHubMonday, November 13th, 2023 at 11:37:54 AM GMT-05:00
GitHubMonday, November 13th, 2023 at 11:39:43 AM GMT-05:00
GitHubMonday, November 13th, 2023 at 11:40:12 AM GMT-05:00
GitHubMonday, November 13th, 2023 at 11:43:26 AM GMT-05:00
GitHubMonday, November 13th, 2023 at 11:45:21 AM GMT-05:00
GitHubMonday, November 13th, 2023 at 2:32:07 PM GMT-05:00
UnknownMonday, November 13th, 2023 at 2:32:08 PM GMT-05:00
GitHubMonday, November 13th, 2023 at 2:34:00 PM GMT-05:00
GitHubTuesday, November 14th, 2023 at 11:14:01 AM GMT-05:00
UnknownThursday, November 16th, 2023 at 9:23:57 AM GMT-05:00
GitHubTuesday, November 14th, 2023 at 6:58:02 PM GMT-05:00
UnknownWednesday, November 15th, 2023 at 9:23:51 AM GMT-05:00
GitHubTuesday, November 14th, 2023 at 7:45:38 PM GMT-05:00
UnknownTuesday, November 14th, 2023 at 7:45:39 PM GMT-05:00
UnknownThursday, November 16th, 2023 at 4:31:48 PM GMT-05:00
UnknownTuesday, November 14th, 2023 at 7:45:39 PM GMT-05:00
GitHubWednesday, November 15th, 2023 at 4:33:43 AM GMT-05:00
UnknownWednesday, November 15th, 2023 at 5:07:16 AM GMT-05:00
GitHubWednesday, November 15th, 2023 at 4:51:28 AM GMT-05:00
UnknownWednesday, November 15th, 2023 at 4:56:42 AM GMT-05:00
UnknownWednesday, November 15th, 2023 at 4:58:47 AM GMT-05:00
GitHubWednesday, November 15th, 2023 at 5:00:01 AM GMT-05:00
UnknownWednesday, November 15th, 2023 at 5:00:01 AM GMT-05:00
UnknownWednesday, November 15th, 2023 at 5:00:01 AM GMT-05:00
GitHubWednesday, November 15th, 2023 at 5:00:32 AM GMT-05:00
UnknownWednesday, November 15th, 2023 at 5:00:33 AM GMT-05:00
UnknownWednesday, November 15th, 2023 at 5:00:33 AM GMT-05:00
GitHubWednesday, November 15th, 2023 at 5:00:50 AM GMT-05:00
UnknownWednesday, November 15th, 2023 at 5:00:51 AM GMT-05:00
UnknownWednesday, November 15th, 2023 at 5:00:51 AM GMT-05:00
GitHubWednesday, November 15th, 2023 at 5:07:16 AM GMT-05:00
UnknownWednesday, November 15th, 2023 at 5:07:16 AM GMT-05:00
GitHubWednesday, November 15th, 2023 at 5:09:46 AM GMT-05:00
GitHubWednesday, November 15th, 2023 at 5:11:47 AM GMT-05:00
GitHubWednesday, November 15th, 2023 at 9:23:51 AM GMT-05:00
UnknownWednesday, November 15th, 2023 at 9:23:51 AM GMT-05:00
GitHubWednesday, November 15th, 2023 at 9:26:45 AM GMT-05:00
GitHubWednesday, November 15th, 2023 at 9:28:49 AM GMT-05:00
GitHubThursday, November 16th, 2023 at 9:23:56 AM GMT-05:00
UnknownThursday, November 16th, 2023 at 9:23:57 AM GMT-05:00
GitHubThursday, November 16th, 2023 at 9:27:31 AM GMT-05:00
GitHubThursday, November 16th, 2023 at 9:29:27 AM GMT-05:00
GitHubThursday, November 16th, 2023 at 4:31:47 PM GMT-05:00
UnknownThursday, November 16th, 2023 at 4:31:48 PM GMT-05:00
GitHubThursday, November 16th, 2023 at 4:34:23 PM GMT-05:00
GitHubThursday, November 16th, 2023 at 4:36:13 PM GMT-05:00
GitHubFriday, November 17th, 2023 at 8:51:38 AM GMT-05:00
GitHubFriday, November 17th, 2023 at 3:11:02 PM GMT-05:00
UnknownFriday, November 17th, 2023 at 4:46:07 PM GMT-05:00
UnknownFriday, November 17th, 2023 at 4:46:07 PM GMT-05:00
GitHubFriday, November 17th, 2023 at 4:51:57 PM GMT-05:00
\ No newline at end of file diff --git a/html/C022MMLU31B-0.html b/html/C022MMLU31B-0.html deleted file mode 100644 index dc919a5..0000000 --- a/html/C022MMLU31B-0.html +++ /dev/null @@ -1,10 +0,0 @@ -Slack

user-generated-metadata

Created by Julien Le Dem on Friday, May 21st, 2021

No messages were ever sent!
\ No newline at end of file diff --git a/html/C04MV3HFU0N-0.html b/html/C04MV3HFU0N-0.html deleted file mode 100644 index ed0afd3..0000000 --- a/html/C04MV3HFU0N-0.html +++ /dev/null @@ -1,6 +0,0 @@ -Slack

github-discussions

Created by Julien Le Dem on Friday, February 3rd, 2023

GitHubSunday, August 20th, 2023 at 12:17:54 AM GMT-04:00
UnknownSunday, September 3rd, 2023 at 12:14:43 AM GMT-04:00
GitHubWednesday, August 23rd, 2023 at 5:15:50 AM GMT-04:00
UnknownWednesday, August 23rd, 2023 at 10:53:27 AM GMT-04:00
GitHubWednesday, August 23rd, 2023 at 5:19:59 AM GMT-04:00
UnknownWednesday, August 23rd, 2023 at 5:19:59 AM GMT-04:00
UnknownWednesday, August 23rd, 2023 at 5:19:59 AM GMT-04:00
UnknownWednesday, August 23rd, 2023 at 10:53:27 AM GMT-04:00
UnknownWednesday, August 23rd, 2023 at 10:53:27 AM GMT-04:00
GitHubWednesday, August 23rd, 2023 at 10:53:27 AM GMT-04:00
GitHubWednesday, August 23rd, 2023 at 1:37:20 PM GMT-04:00
UnknownWednesday, August 23rd, 2023 at 2:35:08 PM GMT-04:00
GitHubWednesday, August 23rd, 2023 at 2:35:07 PM GMT-04:00
UnknownWednesday, August 23rd, 2023 at 2:35:08 PM GMT-04:00
GitHubWednesday, August 23rd, 2023 at 2:36:37 PM GMT-04:00
GitHubWednesday, August 23rd, 2023 at 3:00:09 PM GMT-04:00
GitHubThursday, August 24th, 2023 at 6:41:40 PM GMT-04:00
GitHubSunday, August 27th, 2023 at 12:14:04 AM GMT-04:00
UnknownMonday, August 28th, 2023 at 2:55:46 AM GMT-04:00
GitHubSunday, August 27th, 2023 at 12:14:15 AM GMT-04:00
UnknownMonday, August 28th, 2023 at 3:09:54 AM GMT-04:00
GitHubSunday, August 27th, 2023 at 12:14:52 AM GMT-04:00
UnknownMonday, August 28th, 2023 at 8:41:27 AM GMT-04:00
GitHubSunday, August 27th, 2023 at 12:56:40 AM GMT-04:00
UnknownMonday, August 28th, 2023 at 2:56:11 AM GMT-04:00
GitHubSunday, August 27th, 2023 at 12:57:35 AM GMT-04:00
UnknownMonday, August 28th, 2023 at 2:55:16 AM GMT-04:00
GitHubSunday, August 27th, 2023 at 12:58:31 AM GMT-04:00
UnknownMonday, August 28th, 2023 at 8:36:21 AM GMT-04:00
GitHubSunday, August 27th, 2023 at 12:58:35 AM GMT-04:00
UnknownMonday, August 28th, 2023 at 3:11:23 AM GMT-04:00
GitHubMonday, August 28th, 2023 at 2:55:15 AM GMT-04:00
UnknownMonday, August 28th, 2023 at 2:55:16 AM GMT-04:00
GitHubMonday, August 28th, 2023 at 2:55:46 AM GMT-04:00
UnknownMonday, August 28th, 2023 at 2:55:46 AM GMT-04:00
UnknownMonday, August 28th, 2023 at 2:56:11 AM GMT-04:00
GitHubMonday, August 28th, 2023 at 3:09:53 AM GMT-04:00
UnknownMonday, August 28th, 2023 at 3:09:54 AM GMT-04:00
GitHubMonday, August 28th, 2023 at 3:11:23 AM GMT-04:00
UnknownMonday, August 28th, 2023 at 3:11:23 AM GMT-04:00
UnknownMonday, August 28th, 2023 at 5:33:42 AM GMT-04:00
GitHubMonday, August 28th, 2023 at 7:47:30 AM GMT-04:00
UnknownMonday, August 28th, 2023 at 7:47:30 AM GMT-04:00
UnknownMonday, August 28th, 2023 at 8:39:00 AM GMT-04:00
UnknownMonday, August 28th, 2023 at 7:47:30 AM GMT-04:00
GitHubMonday, August 28th, 2023 at 8:36:21 AM GMT-04:00
UnknownMonday, August 28th, 2023 at 8:36:21 AM GMT-04:00
GitHubMonday, August 28th, 2023 at 8:39:00 AM GMT-04:00
UnknownMonday, August 28th, 2023 at 8:39:00 AM GMT-04:00
UnknownMonday, August 28th, 2023 at 8:39:43 AM GMT-04:00
GitHubMonday, August 28th, 2023 at 8:41:26 AM GMT-04:00
UnknownMonday, August 28th, 2023 at 8:41:27 AM GMT-04:00
GitHubMonday, August 28th, 2023 at 9:39:40 AM GMT-04:00
UnknownMonday, August 28th, 2023 at 10:22:42 AM GMT-04:00
GitHubMonday, August 28th, 2023 at 10:22:41 AM GMT-04:00
UnknownMonday, August 28th, 2023 at 10:22:42 AM GMT-04:00
GitHubTuesday, August 29th, 2023 at 1:45:20 AM GMT-04:00
UnknownWednesday, August 30th, 2023 at 4:18:06 PM GMT-04:00
GitHubWednesday, August 30th, 2023 at 6:22:00 AM GMT-04:00
GitHubWednesday, August 30th, 2023 at 9:07:52 AM GMT-04:00
UnknownWednesday, August 30th, 2023 at 9:07:52 AM GMT-04:00
UnknownMonday, September 4th, 2023 at 10:15:19 AM GMT-04:00
UnknownWednesday, August 30th, 2023 at 9:07:52 AM GMT-04:00
UnknownWednesday, August 30th, 2023 at 9:33:43 AM GMT-04:00
GitHubWednesday, August 30th, 2023 at 9:33:43 AM GMT-04:00
UnknownWednesday, August 30th, 2023 at 9:33:43 AM GMT-04:00
UnknownWednesday, August 30th, 2023 at 4:18:06 PM GMT-04:00
GitHubSaturday, September 2nd, 2023 at 6:19:52 AM GMT-04:00
UnknownSaturday, September 2nd, 2023 at 6:19:53 AM GMT-04:00
UnknownSaturday, September 2nd, 2023 at 6:19:53 AM GMT-04:00
GitHubSunday, September 3rd, 2023 at 12:14:35 AM GMT-04:00
UnknownMonday, September 4th, 2023 at 1:59:12 AM GMT-04:00
GitHubSunday, September 3rd, 2023 at 12:14:39 AM GMT-04:00
UnknownMonday, September 4th, 2023 at 2:01:23 AM GMT-04:00
UnknownSunday, September 3rd, 2023 at 12:14:43 AM GMT-04:00
GitHubSunday, September 3rd, 2023 at 12:25:46 AM GMT-04:00
UnknownMonday, September 4th, 2023 at 2:00:00 AM GMT-04:00
GitHubSunday, September 3rd, 2023 at 12:25:49 AM GMT-04:00
UnknownMonday, September 4th, 2023 at 1:58:45 AM GMT-04:00
GitHubSunday, September 3rd, 2023 at 12:55:45 AM GMT-04:00
UnknownMonday, September 4th, 2023 at 4:57:44 PM GMT-04:00
GitHubMonday, September 4th, 2023 at 1:58:45 AM GMT-04:00
UnknownMonday, September 4th, 2023 at 1:58:45 AM GMT-04:00
UnknownMonday, September 4th, 2023 at 1:59:12 AM GMT-04:00
GitHubMonday, September 4th, 2023 at 1:59:12 AM GMT-04:00
GitHubMonday, September 4th, 2023 at 1:59:59 AM GMT-04:00
UnknownMonday, September 4th, 2023 at 2:00:00 AM GMT-04:00
UnknownMonday, September 4th, 2023 at 2:01:23 AM GMT-04:00
GitHubMonday, September 4th, 2023 at 6:35:02 AM GMT-04:00
GitHubMonday, September 4th, 2023 at 10:15:18 AM GMT-04:00
UnknownMonday, September 4th, 2023 at 10:15:19 AM GMT-04:00
UnknownMonday, September 4th, 2023 at 10:15:20 AM GMT-04:00
GitHubMonday, September 4th, 2023 at 11:22:11 AM GMT-04:00
UnknownMonday, September 4th, 2023 at 11:22:12 AM GMT-04:00
GitHubMonday, September 4th, 2023 at 4:57:44 PM GMT-04:00
UnknownMonday, September 4th, 2023 at 4:57:44 PM GMT-04:00
GitHubTuesday, September 5th, 2023 at 2:27:59 AM GMT-04:00
UnknownTuesday, September 5th, 2023 at 2:27:59 AM GMT-04:00
UnknownFriday, September 8th, 2023 at 9:14:58 AM GMT-04:00
UnknownTuesday, September 5th, 2023 at 2:27:59 AM GMT-04:00
GitHubTuesday, September 5th, 2023 at 3:52:57 AM GMT-04:00
GitHubTuesday, September 5th, 2023 at 4:26:27 AM GMT-04:00
GitHubWednesday, September 6th, 2023 at 4:46:14 AM GMT-04:00
UnknownWednesday, September 6th, 2023 at 4:46:14 AM GMT-04:00
GitHubWednesday, September 6th, 2023 at 5:23:31 AM GMT-04:00
UnknownWednesday, September 6th, 2023 at 5:23:31 AM GMT-04:00
UnknownWednesday, September 6th, 2023 at 5:23:31 AM GMT-04:00
GitHubWednesday, September 6th, 2023 at 4:16:46 PM GMT-04:00
UnknownWednesday, September 6th, 2023 at 4:31:06 PM GMT-04:00
UnknownWednesday, September 6th, 2023 at 4:31:06 PM GMT-04:00
GitHubWednesday, September 6th, 2023 at 4:44:05 PM GMT-04:00
GitHubThursday, September 7th, 2023 at 3:09:06 PM GMT-04:00
UnknownFriday, September 8th, 2023 at 7:20:04 AM GMT-04:00
GitHubFriday, September 8th, 2023 at 3:02:07 AM GMT-04:00
UnknownFriday, September 8th, 2023 at 3:02:07 AM GMT-04:00
UnknownMonday, September 11th, 2023 at 7:20:06 AM GMT-04:00
UnknownFriday, September 8th, 2023 at 3:02:07 AM GMT-04:00
GitHubFriday, September 8th, 2023 at 4:20:09 AM GMT-04:00
UnknownFriday, September 8th, 2023 at 6:18:48 AM GMT-04:00
GitHubFriday, September 8th, 2023 at 6:18:48 AM GMT-04:00
UnknownFriday, September 8th, 2023 at 6:18:48 AM GMT-04:00
GitHubFriday, September 8th, 2023 at 7:20:04 AM GMT-04:00
UnknownFriday, September 8th, 2023 at 7:20:04 AM GMT-04:00
GitHubFriday, September 8th, 2023 at 9:14:58 AM GMT-04:00
UnknownFriday, September 8th, 2023 at 9:14:58 AM GMT-04:00
UnknownFriday, September 8th, 2023 at 9:14:58 AM GMT-04:00
GitHubFriday, September 8th, 2023 at 9:23:26 AM GMT-04:00
UnknownFriday, September 8th, 2023 at 1:27:13 PM GMT-04:00
GitHubFriday, September 8th, 2023 at 1:27:13 PM GMT-04:00
UnknownFriday, September 8th, 2023 at 1:27:13 PM GMT-04:00
GitHubFriday, September 8th, 2023 at 4:34:36 PM GMT-04:00
UnknownThursday, September 14th, 2023 at 11:02:48 AM GMT-04:00
GitHubSunday, September 10th, 2023 at 12:13:54 AM GMT-04:00
UnknownThursday, September 28th, 2023 at 4:25:39 AM GMT-04:00
GitHubSunday, September 10th, 2023 at 12:32:47 AM GMT-04:00
UnknownMonday, September 11th, 2023 at 2:14:33 AM GMT-04:00
GitHubSunday, September 10th, 2023 at 12:32:48 AM GMT-04:00
UnknownMonday, September 11th, 2023 at 2:15:06 AM GMT-04:00
GitHubSunday, September 10th, 2023 at 12:38:44 AM GMT-04:00
UnknownTuesday, September 12th, 2023 at 7:55:00 AM GMT-04:00
UnknownMonday, September 11th, 2023 at 2:14:33 AM GMT-04:00
GitHubMonday, September 11th, 2023 at 2:15:06 AM GMT-04:00
UnknownMonday, September 11th, 2023 at 2:15:06 AM GMT-04:00
GitHubMonday, September 11th, 2023 at 4:30:26 AM GMT-04:00
UnknownTuesday, September 12th, 2023 at 6:59:17 AM GMT-04:00
GitHubMonday, September 11th, 2023 at 5:12:27 AM GMT-04:00
UnknownTuesday, September 12th, 2023 at 6:59:17 AM GMT-04:00
GitHubMonday, September 11th, 2023 at 7:20:04 AM GMT-04:00
GitHubMonday, September 11th, 2023 at 7:20:05 AM GMT-04:00
UnknownMonday, September 11th, 2023 at 7:20:05 AM GMT-04:00
UnknownMonday, September 11th, 2023 at 7:20:05 AM GMT-04:00
UnknownMonday, September 11th, 2023 at 7:20:06 AM GMT-04:00
GitHubMonday, September 11th, 2023 at 8:52:20 AM GMT-04:00
UnknownMonday, September 11th, 2023 at 8:52:21 AM GMT-04:00
UnknownMonday, September 11th, 2023 at 8:52:21 AM GMT-04:00
GitHubMonday, September 11th, 2023 at 5:35:24 PM GMT-04:00
UnknownTuesday, September 12th, 2023 at 9:02:38 AM GMT-04:00
GitHubTuesday, September 12th, 2023 at 5:17:14 AM GMT-04:00
UnknownTuesday, September 12th, 2023 at 5:17:15 AM GMT-04:00
UnknownTuesday, September 12th, 2023 at 5:23:09 AM GMT-04:00
UnknownTuesday, September 12th, 2023 at 5:17:15 AM GMT-04:00
GitHubTuesday, September 12th, 2023 at 5:23:09 AM GMT-04:00
UnknownTuesday, September 12th, 2023 at 5:23:09 AM GMT-04:00
GitHubTuesday, September 12th, 2023 at 6:59:17 AM GMT-04:00
UnknownTuesday, September 12th, 2023 at 6:59:17 AM GMT-04:00
UnknownTuesday, September 12th, 2023 at 6:59:17 AM GMT-04:00
UnknownTuesday, September 12th, 2023 at 7:55:00 AM GMT-04:00
GitHubTuesday, September 12th, 2023 at 9:02:38 AM GMT-04:00
UnknownTuesday, September 12th, 2023 at 9:02:38 AM GMT-04:00
GitHubWednesday, September 13th, 2023 at 2:22:00 AM GMT-04:00
UnknownWednesday, September 13th, 2023 at 2:22:00 AM GMT-04:00
UnknownWednesday, September 13th, 2023 at 5:15:25 AM GMT-04:00
UnknownWednesday, September 13th, 2023 at 2:22:00 AM GMT-04:00
GitHubWednesday, September 13th, 2023 at 5:15:25 AM GMT-04:00
UnknownWednesday, September 13th, 2023 at 5:15:25 AM GMT-04:00
GitHubWednesday, September 13th, 2023 at 7:54:43 AM GMT-04:00
GitHubWednesday, September 13th, 2023 at 3:25:55 PM GMT-04:00
GitHubThursday, September 14th, 2023 at 7:10:43 AM GMT-04:00
UnknownThursday, September 14th, 2023 at 11:02:48 AM GMT-04:00
GitHubThursday, September 14th, 2023 at 8:27:36 AM GMT-04:00
GitHubThursday, September 14th, 2023 at 9:26:32 AM GMT-04:00
UnknownThursday, September 14th, 2023 at 11:03:43 AM GMT-04:00
GitHubThursday, September 14th, 2023 at 9:54:31 AM GMT-04:00
GitHubThursday, September 14th, 2023 at 11:02:47 AM GMT-04:00
UnknownThursday, September 14th, 2023 at 11:02:48 AM GMT-04:00
UnknownThursday, September 14th, 2023 at 11:02:48 AM GMT-04:00
GitHubThursday, September 14th, 2023 at 11:03:42 AM GMT-04:00
UnknownThursday, September 14th, 2023 at 11:03:43 AM GMT-04:00
GitHubThursday, September 14th, 2023 at 11:29:49 AM GMT-04:00
GitHubThursday, September 14th, 2023 at 11:56:53 AM GMT-04:00
UnknownThursday, September 14th, 2023 at 12:18:21 PM GMT-04:00
GitHubThursday, September 14th, 2023 at 12:18:20 PM GMT-04:00
UnknownThursday, September 14th, 2023 at 12:18:21 PM GMT-04:00
GitHubThursday, September 14th, 2023 at 12:27:33 PM GMT-04:00
UnknownThursday, September 14th, 2023 at 12:32:38 PM GMT-04:00
GitHubThursday, September 14th, 2023 at 12:32:38 PM GMT-04:00
UnknownThursday, September 14th, 2023 at 12:32:38 PM GMT-04:00
GitHubThursday, September 14th, 2023 at 12:34:00 PM GMT-04:00
GitHubThursday, September 14th, 2023 at 2:22:05 PM GMT-04:00
UnknownTuesday, September 19th, 2023 at 6:22:34 PM GMT-04:00
GitHubFriday, September 15th, 2023 at 2:36:54 AM GMT-04:00
UnknownFriday, September 15th, 2023 at 2:36:54 AM GMT-04:00
UnknownFriday, September 15th, 2023 at 4:38:27 AM GMT-04:00
UnknownFriday, September 15th, 2023 at 2:36:54 AM GMT-04:00
GitHubFriday, September 15th, 2023 at 4:38:27 AM GMT-04:00
UnknownFriday, September 15th, 2023 at 4:38:27 AM GMT-04:00
GitHubSunday, September 17th, 2023 at 12:20:03 AM GMT-04:00
UnknownWednesday, September 20th, 2023 at 10:57:05 AM GMT-04:00
GitHubSunday, September 17th, 2023 at 12:20:08 AM GMT-04:00
UnknownTuesday, September 26th, 2023 at 8:41:23 AM GMT-04:00
GitHubSunday, September 17th, 2023 at 12:46:03 AM GMT-04:00
UnknownSunday, September 24th, 2023 at 7:42:33 PM GMT-04:00
GitHubTuesday, September 19th, 2023 at 5:42:21 PM GMT-04:00
UnknownTuesday, September 19th, 2023 at 5:52:33 PM GMT-04:00
GitHubTuesday, September 19th, 2023 at 5:52:33 PM GMT-04:00
UnknownTuesday, September 19th, 2023 at 5:52:33 PM GMT-04:00
GitHubTuesday, September 19th, 2023 at 5:54:31 PM GMT-04:00
GitHubTuesday, September 19th, 2023 at 6:22:32 PM GMT-04:00
UnknownTuesday, September 19th, 2023 at 6:22:34 PM GMT-04:00
GitHubTuesday, September 19th, 2023 at 6:27:26 PM GMT-04:00
UnknownTuesday, September 19th, 2023 at 6:36:38 PM GMT-04:00
GitHubTuesday, September 19th, 2023 at 6:36:37 PM GMT-04:00
UnknownTuesday, September 19th, 2023 at 6:36:38 PM GMT-04:00
UnknownWednesday, September 20th, 2023 at 10:57:05 AM GMT-04:00
GitHubWednesday, September 20th, 2023 at 11:28:22 AM GMT-04:00
GitHubWednesday, September 20th, 2023 at 2:56:40 PM GMT-04:00
GitHubWednesday, September 20th, 2023 at 4:09:10 PM GMT-04:00
GitHubFriday, September 22nd, 2023 at 1:36:02 AM GMT-04:00
GitHubFriday, September 22nd, 2023 at 5:34:53 AM GMT-04:00
UnknownFriday, September 22nd, 2023 at 7:01:25 AM GMT-04:00
GitHubFriday, September 22nd, 2023 at 7:01:24 AM GMT-04:00
UnknownFriday, September 22nd, 2023 at 7:01:25 AM GMT-04:00
GitHubSaturday, September 23rd, 2023 at 3:55:04 PM GMT-04:00
GitHubSunday, September 24th, 2023 at 12:10:59 AM GMT-04:00
UnknownTuesday, September 26th, 2023 at 2:38:06 AM GMT-04:00
GitHubSunday, September 24th, 2023 at 12:17:59 AM GMT-04:00
UnknownTuesday, September 26th, 2023 at 2:41:10 AM GMT-04:00
GitHubSunday, September 24th, 2023 at 12:18:06 AM GMT-04:00
UnknownTuesday, September 26th, 2023 at 2:41:45 AM GMT-04:00
GitHubSunday, September 24th, 2023 at 12:24:10 AM GMT-04:00
UnknownTuesday, September 26th, 2023 at 2:38:31 AM GMT-04:00
UnknownSunday, September 24th, 2023 at 7:42:33 PM GMT-04:00
GitHubMonday, September 25th, 2023 at 11:01:11 AM GMT-04:00
UnknownThursday, September 28th, 2023 at 7:50:07 AM GMT-04:00
GitHubMonday, September 25th, 2023 at 11:14:17 AM GMT-04:00
UnknownThursday, September 28th, 2023 at 7:51:57 AM GMT-04:00
GitHubTuesday, September 26th, 2023 at 2:24:32 AM GMT-04:00
UnknownWednesday, October 25th, 2023 at 3:34:22 AM GMT-04:00
GitHubTuesday, September 26th, 2023 at 2:38:06 AM GMT-04:00
UnknownTuesday, September 26th, 2023 at 2:38:06 AM GMT-04:00
UnknownTuesday, September 26th, 2023 at 2:38:31 AM GMT-04:00
GitHubTuesday, September 26th, 2023 at 2:38:31 AM GMT-04:00
GitHubTuesday, September 26th, 2023 at 2:41:08 AM GMT-04:00
UnknownTuesday, September 26th, 2023 at 2:41:10 AM GMT-04:00
GitHubTuesday, September 26th, 2023 at 2:41:45 AM GMT-04:00
UnknownTuesday, September 26th, 2023 at 2:41:45 AM GMT-04:00
GitHubTuesday, September 26th, 2023 at 7:22:25 AM GMT-04:00
UnknownTuesday, September 26th, 2023 at 7:22:26 AM GMT-04:00
UnknownTuesday, September 26th, 2023 at 7:40:05 AM GMT-04:00
UnknownTuesday, September 26th, 2023 at 7:22:26 AM GMT-04:00
GitHubTuesday, September 26th, 2023 at 7:40:04 AM GMT-04:00
UnknownTuesday, September 26th, 2023 at 7:40:05 AM GMT-04:00
GitHubTuesday, September 26th, 2023 at 8:00:33 AM GMT-04:00
UnknownTuesday, September 26th, 2023 at 8:00:33 AM GMT-04:00
UnknownTuesday, September 26th, 2023 at 8:00:33 AM GMT-04:00
GitHubTuesday, September 26th, 2023 at 8:39:47 AM GMT-04:00
UnknownTuesday, September 26th, 2023 at 8:39:47 AM GMT-04:00
UnknownTuesday, September 26th, 2023 at 8:39:47 AM GMT-04:00
GitHubTuesday, September 26th, 2023 at 8:41:23 AM GMT-04:00
UnknownTuesday, September 26th, 2023 at 8:41:23 AM GMT-04:00
GitHubWednesday, September 27th, 2023 at 3:07:11 AM GMT-04:00
GitHubWednesday, September 27th, 2023 at 8:19:41 AM GMT-04:00
GitHubWednesday, September 27th, 2023 at 8:19:47 AM GMT-04:00
UnknownWednesday, September 27th, 2023 at 8:19:47 AM GMT-04:00
UnknownWednesday, September 27th, 2023 at 8:19:47 AM GMT-04:00
GitHubWednesday, September 27th, 2023 at 8:32:12 AM GMT-04:00
UnknownWednesday, September 27th, 2023 at 8:32:13 AM GMT-04:00
UnknownMonday, October 16th, 2023 at 4:30:59 AM GMT-04:00
UnknownWednesday, September 27th, 2023 at 8:32:13 AM GMT-04:00
GitHubWednesday, September 27th, 2023 at 9:02:53 AM GMT-04:00
UnknownFriday, October 6th, 2023 at 7:33:42 AM GMT-04:00
GitHubWednesday, September 27th, 2023 at 10:58:54 AM GMT-04:00
UnknownThursday, September 28th, 2023 at 7:50:05 AM GMT-04:00
GitHubWednesday, September 27th, 2023 at 12:43:15 PM GMT-04:00
GitHubThursday, September 28th, 2023 at 4:25:39 AM GMT-04:00
UnknownThursday, September 28th, 2023 at 4:25:39 AM GMT-04:00
GitHubThursday, September 28th, 2023 at 4:42:07 AM GMT-04:00
UnknownThursday, September 28th, 2023 at 7:51:57 AM GMT-04:00
GitHubThursday, September 28th, 2023 at 7:38:48 AM GMT-04:00
UnknownFriday, September 29th, 2023 at 9:20:27 AM GMT-04:00
UnknownThursday, September 28th, 2023 at 7:50:05 AM GMT-04:00
GitHubThursday, September 28th, 2023 at 7:50:06 AM GMT-04:00
UnknownThursday, September 28th, 2023 at 7:50:07 AM GMT-04:00
GitHubThursday, September 28th, 2023 at 7:51:56 AM GMT-04:00
UnknownThursday, September 28th, 2023 at 7:51:57 AM GMT-04:00
UnknownThursday, September 28th, 2023 at 7:51:57 AM GMT-04:00
GitHubThursday, September 28th, 2023 at 11:38:12 AM GMT-04:00
UnknownThursday, September 28th, 2023 at 12:02:27 PM GMT-04:00
UnknownThursday, September 28th, 2023 at 12:02:27 PM GMT-04:00
GitHubThursday, September 28th, 2023 at 12:02:27 PM GMT-04:00
GitHubThursday, September 28th, 2023 at 6:13:11 PM GMT-04:00
UnknownFriday, September 29th, 2023 at 4:43:53 AM GMT-04:00
GitHubFriday, September 29th, 2023 at 4:43:52 AM GMT-04:00
UnknownFriday, September 29th, 2023 at 4:43:53 AM GMT-04:00
GitHubFriday, September 29th, 2023 at 5:12:10 AM GMT-04:00
UnknownMonday, October 2nd, 2023 at 2:13:19 AM GMT-04:00
GitHubFriday, September 29th, 2023 at 7:11:10 AM GMT-04:00
UnknownFriday, September 29th, 2023 at 7:11:11 AM GMT-04:00
UnknownMonday, October 2nd, 2023 at 4:21:45 AM GMT-04:00
UnknownFriday, September 29th, 2023 at 7:11:11 AM GMT-04:00
GitHubFriday, September 29th, 2023 at 9:20:26 AM GMT-04:00
UnknownFriday, September 29th, 2023 at 9:20:27 AM GMT-04:00
GitHubFriday, September 29th, 2023 at 10:23:18 AM GMT-04:00
GitHubSunday, October 1st, 2023 at 12:34:14 AM GMT-04:00
UnknownMonday, October 2nd, 2023 at 2:14:36 AM GMT-04:00
GitHubSunday, October 1st, 2023 at 12:34:25 AM GMT-04:00
UnknownMonday, October 2nd, 2023 at 2:14:11 AM GMT-04:00
GitHubSunday, October 1st, 2023 at 12:51:58 AM GMT-04:00
UnknownSunday, October 1st, 2023 at 1:36:56 AM GMT-04:00
UnknownSunday, October 1st, 2023 at 1:36:56 AM GMT-04:00
GitHubSunday, October 1st, 2023 at 1:36:57 AM GMT-04:00
GitHubMonday, October 2nd, 2023 at 2:13:19 AM GMT-04:00
UnknownMonday, October 2nd, 2023 at 2:13:19 AM GMT-04:00
GitHubMonday, October 2nd, 2023 at 2:14:10 AM GMT-04:00
UnknownMonday, October 2nd, 2023 at 2:14:11 AM GMT-04:00
GitHubMonday, October 2nd, 2023 at 2:14:36 AM GMT-04:00
UnknownMonday, October 2nd, 2023 at 2:14:36 AM GMT-04:00
GitHubMonday, October 2nd, 2023 at 3:05:39 AM GMT-04:00
UnknownMonday, October 2nd, 2023 at 3:05:39 AM GMT-04:00
UnknownMonday, October 2nd, 2023 at 3:05:39 AM GMT-04:00
UnknownMonday, October 2nd, 2023 at 4:21:45 AM GMT-04:00
GitHubMonday, October 2nd, 2023 at 4:21:45 AM GMT-04:00
GitHubMonday, October 2nd, 2023 at 2:37:37 PM GMT-04:00
UnknownTuesday, October 3rd, 2023 at 9:02:00 AM GMT-04:00
GitHubMonday, October 2nd, 2023 at 9:48:34 PM GMT-04:00
UnknownWednesday, October 4th, 2023 at 2:16:58 AM GMT-04:00
GitHubTuesday, October 3rd, 2023 at 8:30:35 AM GMT-04:00
GitHubTuesday, October 3rd, 2023 at 8:49:45 AM GMT-04:00
UnknownWednesday, October 4th, 2023 at 3:33:43 AM GMT-04:00
GitHubTuesday, October 3rd, 2023 at 9:02:00 AM GMT-04:00
UnknownTuesday, October 3rd, 2023 at 9:02:00 AM GMT-04:00
GitHubTuesday, October 3rd, 2023 at 9:03:12 AM GMT-04:00
GitHubTuesday, October 3rd, 2023 at 9:13:01 AM GMT-04:00
UnknownTuesday, October 3rd, 2023 at 10:07:58 AM GMT-04:00
GitHubTuesday, October 3rd, 2023 at 10:07:57 AM GMT-04:00
UnknownTuesday, October 3rd, 2023 at 10:07:58 AM GMT-04:00
GitHubTuesday, October 3rd, 2023 at 10:08:47 AM GMT-04:00
UnknownWednesday, October 4th, 2023 at 3:17:15 AM GMT-04:00
GitHubTuesday, October 3rd, 2023 at 10:08:49 AM GMT-04:00
UnknownWednesday, October 4th, 2023 at 2:41:11 AM GMT-04:00
GitHubTuesday, October 3rd, 2023 at 10:08:52 AM GMT-04:00
UnknownThursday, October 5th, 2023 at 4:13:10 PM GMT-04:00
GitHubTuesday, October 3rd, 2023 at 10:08:59 AM GMT-04:00
UnknownWednesday, October 4th, 2023 at 4:30:56 AM GMT-04:00
GitHubTuesday, October 3rd, 2023 at 10:09:15 AM GMT-04:00
GitHubTuesday, October 3rd, 2023 at 10:09:24 AM GMT-04:00
UnknownWednesday, October 4th, 2023 at 2:43:39 AM GMT-04:00
GitHubTuesday, October 3rd, 2023 at 10:09:30 AM GMT-04:00
UnknownWednesday, October 4th, 2023 at 2:43:00 AM GMT-04:00
GitHubTuesday, October 3rd, 2023 at 10:33:29 AM GMT-04:00
UnknownWednesday, October 4th, 2023 at 2:16:58 AM GMT-04:00
GitHubWednesday, October 4th, 2023 at 2:16:58 AM GMT-04:00
GitHubWednesday, October 4th, 2023 at 2:41:10 AM GMT-04:00
UnknownWednesday, October 4th, 2023 at 2:41:11 AM GMT-04:00
UnknownWednesday, October 4th, 2023 at 2:43:00 AM GMT-04:00
GitHubWednesday, October 4th, 2023 at 2:43:38 AM GMT-04:00
UnknownWednesday, October 4th, 2023 at 2:43:39 AM GMT-04:00
UnknownWednesday, October 4th, 2023 at 3:17:15 AM GMT-04:00
GitHubWednesday, October 4th, 2023 at 3:33:42 AM GMT-04:00
UnknownWednesday, October 4th, 2023 at 3:33:43 AM GMT-04:00
UnknownWednesday, October 4th, 2023 at 4:30:56 AM GMT-04:00
GitHubWednesday, October 4th, 2023 at 4:30:57 AM GMT-04:00
GitHubWednesday, October 4th, 2023 at 8:48:58 AM GMT-04:00
UnknownWednesday, October 4th, 2023 at 8:48:59 AM GMT-04:00
UnknownFriday, October 6th, 2023 at 7:33:42 AM GMT-04:00
UnknownWednesday, October 4th, 2023 at 8:48:59 AM GMT-04:00
GitHubWednesday, October 4th, 2023 at 10:57:04 AM GMT-04:00
GitHubWednesday, October 4th, 2023 at 9:48:11 PM GMT-04:00
GitHubThursday, October 5th, 2023 at 4:13:10 PM GMT-04:00
UnknownThursday, October 5th, 2023 at 4:13:10 PM GMT-04:00
GitHubThursday, October 5th, 2023 at 5:27:43 PM GMT-04:00
GitHubThursday, October 5th, 2023 at 5:50:31 PM GMT-04:00
GitHubThursday, October 5th, 2023 at 7:06:12 PM GMT-04:00
GitHubFriday, October 6th, 2023 at 4:29:35 AM GMT-04:00
GitHubFriday, October 6th, 2023 at 7:33:42 AM GMT-04:00
UnknownFriday, October 6th, 2023 at 7:33:42 AM GMT-04:00
UnknownFriday, October 6th, 2023 at 7:33:42 AM GMT-04:00
GitHubFriday, October 6th, 2023 at 8:26:48 AM GMT-04:00
UnknownMonday, October 9th, 2023 at 4:35:46 AM GMT-04:00
GitHubFriday, October 6th, 2023 at 9:18:00 AM GMT-04:00
UnknownFriday, October 20th, 2023 at 7:05:52 AM GMT-04:00
GitHubSunday, October 8th, 2023 at 12:19:12 AM GMT-04:00
UnknownMonday, October 9th, 2023 at 2:23:18 AM GMT-04:00
GitHubSunday, October 8th, 2023 at 12:19:31 AM GMT-04:00
UnknownMonday, October 9th, 2023 at 2:23:47 AM GMT-04:00
GitHubSunday, October 8th, 2023 at 12:19:37 AM GMT-04:00
UnknownMonday, October 9th, 2023 at 2:23:37 AM GMT-04:00
GitHubSunday, October 8th, 2023 at 12:19:39 AM GMT-04:00
UnknownMonday, October 9th, 2023 at 3:28:20 AM GMT-04:00
GitHubSunday, October 8th, 2023 at 12:39:35 AM GMT-04:00
UnknownMonday, October 9th, 2023 at 2:20:49 AM GMT-04:00
GitHubSunday, October 8th, 2023 at 12:40:36 AM GMT-04:00
UnknownMonday, October 9th, 2023 at 4:36:40 AM GMT-04:00
GitHubSunday, October 8th, 2023 at 12:41:33 AM GMT-04:00
UnknownMonday, October 9th, 2023 at 2:21:15 AM GMT-04:00
GitHubMonday, October 9th, 2023 at 12:16:01 AM GMT-04:00
UnknownThursday, October 19th, 2023 at 2:20:44 AM GMT-04:00
UnknownMonday, October 9th, 2023 at 2:20:49 AM GMT-04:00
GitHubMonday, October 9th, 2023 at 2:20:50 AM GMT-04:00
UnknownMonday, October 9th, 2023 at 2:21:15 AM GMT-04:00
GitHubMonday, October 9th, 2023 at 2:21:16 AM GMT-04:00
UnknownMonday, October 9th, 2023 at 2:23:18 AM GMT-04:00
UnknownMonday, October 9th, 2023 at 2:23:37 AM GMT-04:00
UnknownMonday, October 9th, 2023 at 2:23:47 AM GMT-04:00
GitHubMonday, October 9th, 2023 at 2:25:30 AM GMT-04:00
UnknownMonday, October 9th, 2023 at 3:28:20 AM GMT-04:00
GitHubMonday, October 9th, 2023 at 3:28:21 AM GMT-04:00
UnknownMonday, October 9th, 2023 at 4:35:46 AM GMT-04:00
GitHubMonday, October 9th, 2023 at 4:35:46 AM GMT-04:00
UnknownMonday, October 9th, 2023 at 4:35:47 AM GMT-04:00
GitHubMonday, October 9th, 2023 at 4:35:47 AM GMT-04:00
UnknownMonday, October 9th, 2023 at 4:35:47 AM GMT-04:00
UnknownMonday, October 9th, 2023 at 4:36:40 AM GMT-04:00
GitHubMonday, October 9th, 2023 at 4:36:40 AM GMT-04:00
GitHubMonday, October 9th, 2023 at 7:54:56 AM GMT-04:00
UnknownThursday, October 12th, 2023 at 4:44:41 AM GMT-04:00
GitHubMonday, October 9th, 2023 at 9:47:28 AM GMT-04:00
UnknownMonday, October 9th, 2023 at 9:58:11 AM GMT-04:00
UnknownMonday, October 9th, 2023 at 9:58:11 AM GMT-04:00
GitHubMonday, October 9th, 2023 at 9:58:12 AM GMT-04:00
GitHubMonday, October 9th, 2023 at 9:59:11 AM GMT-04:00
GitHubMonday, October 9th, 2023 at 1:33:03 PM GMT-04:00
UnknownMonday, October 9th, 2023 at 2:32:44 PM GMT-04:00
UnknownMonday, October 9th, 2023 at 2:32:44 PM GMT-04:00
GitHubMonday, October 9th, 2023 at 2:32:45 PM GMT-04:00
GitHubMonday, October 9th, 2023 at 2:37:37 PM GMT-04:00
UnknownMonday, October 9th, 2023 at 2:42:15 PM GMT-04:00
UnknownMonday, October 9th, 2023 at 2:42:15 PM GMT-04:00
GitHubMonday, October 9th, 2023 at 2:42:15 PM GMT-04:00
GitHubMonday, October 9th, 2023 at 2:43:19 PM GMT-04:00
GitHubMonday, October 9th, 2023 at 3:16:16 PM GMT-04:00
GitHubTuesday, October 10th, 2023 at 7:55:33 AM GMT-04:00
UnknownTuesday, October 10th, 2023 at 7:55:33 AM GMT-04:00
UnknownWednesday, October 11th, 2023 at 2:04:28 AM GMT-04:00
UnknownTuesday, October 10th, 2023 at 7:55:33 AM GMT-04:00
UnknownWednesday, October 11th, 2023 at 2:04:28 AM GMT-04:00
GitHubWednesday, October 11th, 2023 at 2:04:28 AM GMT-04:00
GitHubWednesday, October 11th, 2023 at 2:17:24 AM GMT-04:00
UnknownThursday, November 9th, 2023 at 10:15:37 AM GMT-05:00
GitHubWednesday, October 11th, 2023 at 6:34:27 AM GMT-04:00
UnknownWednesday, October 11th, 2023 at 6:34:27 AM GMT-04:00
UnknownMonday, October 16th, 2023 at 2:48:04 AM GMT-04:00
UnknownWednesday, October 11th, 2023 at 6:34:27 AM GMT-04:00
UnknownThursday, October 12th, 2023 at 4:44:41 AM GMT-04:00
GitHubThursday, October 12th, 2023 at 4:44:41 AM GMT-04:00
GitHubFriday, October 13th, 2023 at 9:14:55 AM GMT-04:00
GitHubFriday, October 13th, 2023 at 9:28:45 AM GMT-04:00
GitHubFriday, October 13th, 2023 at 1:03:14 PM GMT-04:00
GitHubSunday, October 15th, 2023 at 12:05:12 AM GMT-04:00
UnknownMonday, October 16th, 2023 at 2:04:25 AM GMT-04:00
GitHubSunday, October 15th, 2023 at 12:05:13 AM GMT-04:00
UnknownMonday, October 16th, 2023 at 2:12:15 AM GMT-04:00
GitHubSunday, October 15th, 2023 at 12:18:47 AM GMT-04:00
UnknownWednesday, October 18th, 2023 at 2:05:19 AM GMT-04:00
GitHubSunday, October 15th, 2023 at 12:42:08 AM GMT-04:00
UnknownMonday, October 16th, 2023 at 2:04:56 AM GMT-04:00
GitHubSunday, October 15th, 2023 at 12:42:16 AM GMT-04:00
UnknownMonday, October 16th, 2023 at 2:05:55 AM GMT-04:00
GitHubSunday, October 15th, 2023 at 12:42:28 AM GMT-04:00
UnknownMonday, October 16th, 2023 at 2:05:25 AM GMT-04:00
GitHubSunday, October 15th, 2023 at 12:52:00 AM GMT-04:00
UnknownMonday, October 16th, 2023 at 2:02:31 AM GMT-04:00
GitHubSunday, October 15th, 2023 at 12:52:01 AM GMT-04:00
UnknownMonday, October 16th, 2023 at 2:06:19 AM GMT-04:00
GitHubMonday, October 16th, 2023 at 12:44:43 AM GMT-04:00
GitHubMonday, October 16th, 2023 at 12:44:54 AM GMT-04:00
UnknownMonday, October 16th, 2023 at 2:02:31 AM GMT-04:00
GitHubMonday, October 16th, 2023 at 2:02:32 AM GMT-04:00
UnknownMonday, October 16th, 2023 at 2:04:25 AM GMT-04:00
GitHubMonday, October 16th, 2023 at 2:04:55 AM GMT-04:00
UnknownMonday, October 16th, 2023 at 2:04:56 AM GMT-04:00
GitHubMonday, October 16th, 2023 at 2:05:25 AM GMT-04:00
UnknownMonday, October 16th, 2023 at 2:05:25 AM GMT-04:00
UnknownMonday, October 16th, 2023 at 2:05:55 AM GMT-04:00
UnknownMonday, October 16th, 2023 at 2:06:19 AM GMT-04:00
UnknownMonday, October 16th, 2023 at 2:12:15 AM GMT-04:00
UnknownMonday, October 16th, 2023 at 2:48:04 AM GMT-04:00
UnknownMonday, October 16th, 2023 at 4:30:59 AM GMT-04:00
GitHubMonday, October 16th, 2023 at 4:30:59 AM GMT-04:00
UnknownMonday, October 16th, 2023 at 4:30:59 AM GMT-04:00
UnknownMonday, October 16th, 2023 at 4:30:59 AM GMT-04:00
GitHubMonday, October 16th, 2023 at 4:31:00 AM GMT-04:00
GitHubMonday, October 16th, 2023 at 8:02:57 AM GMT-04:00
UnknownMonday, October 16th, 2023 at 8:02:58 AM GMT-04:00
UnknownMonday, October 23rd, 2023 at 5:57:30 AM GMT-04:00
UnknownMonday, October 16th, 2023 at 8:02:58 AM GMT-04:00
GitHubTuesday, October 17th, 2023 at 4:35:32 AM GMT-04:00
UnknownTuesday, October 17th, 2023 at 4:35:33 AM GMT-04:00
UnknownWednesday, October 25th, 2023 at 3:34:22 AM GMT-04:00
UnknownTuesday, October 17th, 2023 at 4:35:33 AM GMT-04:00
UnknownWednesday, October 18th, 2023 at 2:05:19 AM GMT-04:00
GitHubWednesday, October 18th, 2023 at 10:29:38 AM GMT-04:00
GitHubThursday, October 19th, 2023 at 2:20:44 AM GMT-04:00
UnknownThursday, October 19th, 2023 at 2:20:44 AM GMT-04:00
GitHubThursday, October 19th, 2023 at 5:43:58 AM GMT-04:00
UnknownThursday, October 19th, 2023 at 5:43:58 AM GMT-04:00
UnknownThursday, October 26th, 2023 at 5:48:16 AM GMT-04:00
UnknownThursday, October 19th, 2023 at 5:43:58 AM GMT-04:00
GitHubThursday, October 19th, 2023 at 8:01:51 AM GMT-04:00
UnknownThursday, October 19th, 2023 at 8:01:51 AM GMT-04:00
UnknownFriday, October 20th, 2023 at 7:05:52 AM GMT-04:00
UnknownThursday, October 19th, 2023 at 8:01:51 AM GMT-04:00
GitHubFriday, October 20th, 2023 at 6:58:41 AM GMT-04:00
UnknownMonday, October 23rd, 2023 at 3:45:25 AM GMT-04:00
GitHubFriday, October 20th, 2023 at 7:05:52 AM GMT-04:00
UnknownFriday, October 20th, 2023 at 7:05:52 AM GMT-04:00
UnknownFriday, October 20th, 2023 at 7:05:52 AM GMT-04:00
GitHubFriday, October 20th, 2023 at 7:49:50 AM GMT-04:00
UnknownMonday, October 23rd, 2023 at 3:46:24 AM GMT-04:00
GitHubFriday, October 20th, 2023 at 10:03:38 AM GMT-04:00
GitHubFriday, October 20th, 2023 at 10:22:06 AM GMT-04:00
GTCSaturday, October 21st, 2023 at 4:59:15 AM GMT-04:00
@GTC has joined the channel
GitHubSunday, October 22nd, 2023 at 12:01:45 AM GMT-04:00
UnknownMonday, October 23rd, 2023 at 3:51:04 AM GMT-04:00
GitHubSunday, October 22nd, 2023 at 12:01:53 AM GMT-04:00
UnknownMonday, October 23rd, 2023 at 3:51:51 AM GMT-04:00
GitHubSunday, October 22nd, 2023 at 12:35:40 AM GMT-04:00
UnknownMonday, October 23rd, 2023 at 3:52:39 AM GMT-04:00
UnknownMonday, October 23rd, 2023 at 3:45:25 AM GMT-04:00
GitHubMonday, October 23rd, 2023 at 3:46:23 AM GMT-04:00
UnknownMonday, October 23rd, 2023 at 3:46:24 AM GMT-04:00
GitHubMonday, October 23rd, 2023 at 3:51:03 AM GMT-04:00
UnknownMonday, October 23rd, 2023 at 3:51:04 AM GMT-04:00
GitHubMonday, October 23rd, 2023 at 3:51:50 AM GMT-04:00
UnknownMonday, October 23rd, 2023 at 3:51:51 AM GMT-04:00
UnknownMonday, October 23rd, 2023 at 3:52:39 AM GMT-04:00
GitHubMonday, October 23rd, 2023 at 3:52:40 AM GMT-04:00
GitHubMonday, October 23rd, 2023 at 5:57:30 AM GMT-04:00
UnknownMonday, October 23rd, 2023 at 5:57:30 AM GMT-04:00
GitHubTuesday, October 24th, 2023 at 7:34:26 PM GMT-04:00
UnknownThursday, October 26th, 2023 at 5:49:47 AM GMT-04:00
GitHubTuesday, October 24th, 2023 at 11:57:46 PM GMT-04:00
UnknownFriday, November 10th, 2023 at 7:19:42 AM GMT-05:00
GitHubWednesday, October 25th, 2023 at 3:34:21 AM GMT-04:00
UnknownWednesday, October 25th, 2023 at 3:34:22 AM GMT-04:00
UnknownWednesday, October 25th, 2023 at 3:34:22 AM GMT-04:00
GitHubWednesday, October 25th, 2023 at 3:40:39 AM GMT-04:00
UnknownThursday, October 26th, 2023 at 5:40:57 AM GMT-04:00
GitHubWednesday, October 25th, 2023 at 5:56:06 PM GMT-04:00
GitHubThursday, October 26th, 2023 at 4:25:10 AM GMT-04:00
UnknownThursday, October 26th, 2023 at 4:25:10 AM GMT-04:00
UnknownThursday, October 26th, 2023 at 5:40:57 AM GMT-04:00
UnknownThursday, October 26th, 2023 at 4:25:10 AM GMT-04:00
GitHubThursday, October 26th, 2023 at 5:40:56 AM GMT-04:00
UnknownThursday, October 26th, 2023 at 5:40:57 AM GMT-04:00
UnknownThursday, October 26th, 2023 at 5:40:57 AM GMT-04:00
GitHubThursday, October 26th, 2023 at 5:48:16 AM GMT-04:00
UnknownThursday, October 26th, 2023 at 5:48:16 AM GMT-04:00
UnknownThursday, October 26th, 2023 at 5:49:47 AM GMT-04:00
GitHubThursday, October 26th, 2023 at 6:16:53 AM GMT-04:00
UnknownThursday, October 26th, 2023 at 6:35:14 AM GMT-04:00
UnknownThursday, October 26th, 2023 at 7:48:49 AM GMT-04:00
UnknownThursday, October 26th, 2023 at 6:35:14 AM GMT-04:00
GitHubThursday, October 26th, 2023 at 7:08:27 AM GMT-04:00
GitHubThursday, October 26th, 2023 at 7:08:27 AM GMT-04:00
UnknownThursday, October 26th, 2023 at 7:08:27 AM GMT-04:00
UnknownThursday, October 26th, 2023 at 7:08:27 AM GMT-04:00
UnknownThursday, October 26th, 2023 at 7:48:49 AM GMT-04:00
GitHubFriday, October 27th, 2023 at 5:00:21 AM GMT-04:00
UnknownWednesday, November 15th, 2023 at 4:58:47 AM GMT-05:00
GitHubFriday, October 27th, 2023 at 9:23:56 AM GMT-04:00
UnknownFriday, October 27th, 2023 at 9:23:56 AM GMT-04:00
UnknownFriday, October 27th, 2023 at 9:33:24 AM GMT-04:00
UnknownFriday, October 27th, 2023 at 9:23:56 AM GMT-04:00
GitHubFriday, October 27th, 2023 at 9:33:23 AM GMT-04:00
UnknownFriday, October 27th, 2023 at 9:33:24 AM GMT-04:00
GitHubSunday, October 29th, 2023 at 12:24:29 AM GMT-04:00
UnknownMonday, October 30th, 2023 at 2:53:06 AM GMT-04:00
GitHubMonday, October 30th, 2023 at 12:45:40 AM GMT-04:00
GitHubMonday, October 30th, 2023 at 2:53:05 AM GMT-04:00
UnknownMonday, October 30th, 2023 at 2:53:06 AM GMT-04:00
GitHubMonday, October 30th, 2023 at 11:17:21 AM GMT-04:00
UnknownMonday, October 30th, 2023 at 5:17:18 PM GMT-04:00
GitHubMonday, October 30th, 2023 at 11:36:04 AM GMT-04:00
UnknownFriday, November 10th, 2023 at 4:35:54 AM GMT-05:00
GitHubMonday, October 30th, 2023 at 5:17:17 PM GMT-04:00
UnknownMonday, October 30th, 2023 at 5:17:18 PM GMT-04:00
GitHubMonday, October 30th, 2023 at 6:49:10 PM GMT-04:00
UnknownTuesday, October 31st, 2023 at 8:36:45 AM GMT-04:00
GitHubTuesday, October 31st, 2023 at 8:36:45 AM GMT-04:00
UnknownTuesday, October 31st, 2023 at 8:36:45 AM GMT-04:00
GitHubTuesday, October 31st, 2023 at 8:06:36 PM GMT-04:00
UnknownWednesday, November 15th, 2023 at 4:56:42 AM GMT-05:00
GitHubWednesday, November 1st, 2023 at 10:06:55 AM GMT-04:00
UnknownWednesday, November 8th, 2023 at 10:16:34 AM GMT-05:00
GitHubWednesday, November 1st, 2023 at 12:28:09 PM GMT-04:00
UnknownThursday, November 2nd, 2023 at 5:42:35 AM GMT-04:00
GitHubThursday, November 2nd, 2023 at 5:42:34 AM GMT-04:00
UnknownThursday, November 2nd, 2023 at 5:42:35 AM GMT-04:00
GitHubThursday, November 2nd, 2023 at 5:46:19 AM GMT-04:00
GitHubThursday, November 2nd, 2023 at 6:10:25 AM GMT-04:00
GitHubFriday, November 3rd, 2023 at 7:31:33 AM GMT-04:00
GitHubFriday, November 3rd, 2023 at 7:43:08 AM GMT-04:00
UnknownThursday, November 9th, 2023 at 5:03:09 AM GMT-05:00
GitHubFriday, November 3rd, 2023 at 10:38:19 AM GMT-04:00
UnknownFriday, November 3rd, 2023 at 2:14:40 PM GMT-04:00
UnknownFriday, November 3rd, 2023 at 2:14:40 PM GMT-04:00
GitHubFriday, November 3rd, 2023 at 4:55:02 PM GMT-04:00
GitHubFriday, November 3rd, 2023 at 6:30:02 PM GMT-04:00
UnknownWednesday, November 8th, 2023 at 7:56:49 AM GMT-05:00
GitHubFriday, November 3rd, 2023 at 6:34:38 PM GMT-04:00
UnknownWednesday, November 8th, 2023 at 7:56:49 AM GMT-05:00
GitHubSunday, November 5th, 2023 at 12:03:50 AM GMT-04:00
UnknownTuesday, November 7th, 2023 at 7:16:58 AM GMT-05:00
GitHubMonday, November 6th, 2023 at 9:20:37 AM GMT-05:00
UnknownMonday, November 13th, 2023 at 4:15:17 AM GMT-05:00
GitHubTuesday, November 7th, 2023 at 7:16:57 AM GMT-05:00
UnknownTuesday, November 7th, 2023 at 7:16:58 AM GMT-05:00
GitHubTuesday, November 7th, 2023 at 10:08:25 AM GMT-05:00
UnknownTuesday, November 7th, 2023 at 10:08:25 AM GMT-05:00
UnknownWednesday, November 8th, 2023 at 1:29:17 AM GMT-05:00
UnknownTuesday, November 7th, 2023 at 10:08:25 AM GMT-05:00
GitHubWednesday, November 8th, 2023 at 1:29:17 AM GMT-05:00
UnknownWednesday, November 8th, 2023 at 1:29:17 AM GMT-05:00
GitHubWednesday, November 8th, 2023 at 2:56:00 AM GMT-05:00
UnknownWednesday, November 8th, 2023 at 2:56:00 AM GMT-05:00
UnknownWednesday, November 8th, 2023 at 6:15:34 AM GMT-05:00
UnknownWednesday, November 8th, 2023 at 2:56:00 AM GMT-05:00
GitHubWednesday, November 8th, 2023 at 3:42:06 AM GMT-05:00
UnknownWednesday, November 8th, 2023 at 4:05:30 AM GMT-05:00
UnknownWednesday, November 8th, 2023 at 4:05:30 AM GMT-05:00
GitHubWednesday, November 8th, 2023 at 4:05:31 AM GMT-05:00
GitHubWednesday, November 8th, 2023 at 6:15:33 AM GMT-05:00
UnknownWednesday, November 8th, 2023 at 6:15:34 AM GMT-05:00
GitHubWednesday, November 8th, 2023 at 7:56:48 AM GMT-05:00
UnknownWednesday, November 8th, 2023 at 7:56:49 AM GMT-05:00
UnknownWednesday, November 8th, 2023 at 7:56:49 AM GMT-05:00
GitHubWednesday, November 8th, 2023 at 10:16:33 AM GMT-05:00
UnknownWednesday, November 8th, 2023 at 10:16:34 AM GMT-05:00
GitHubWednesday, November 8th, 2023 at 4:32:47 PM GMT-05:00
UnknownThursday, November 9th, 2023 at 5:08:05 AM GMT-05:00
GitHubThursday, November 9th, 2023 at 5:03:08 AM GMT-05:00
UnknownThursday, November 9th, 2023 at 5:03:09 AM GMT-05:00
GitHubThursday, November 9th, 2023 at 5:08:05 AM GMT-05:00
UnknownThursday, November 9th, 2023 at 5:08:05 AM GMT-05:00
UnknownThursday, November 9th, 2023 at 10:15:37 AM GMT-05:00
GitHubThursday, November 9th, 2023 at 3:29:23 PM GMT-05:00
Sheeri Cabral (Collibra)Thursday, November 9th, 2023 at 3:30:23 PM GMT-05:00
@Sheeri Cabral (Collibra) has joined the channel
Sheeri Cabral (Collibra)Thursday, November 9th, 2023 at 3:30:58 PM GMT-05:00
Let me know if I did that wrong ^^^ It’s been a while since I modified someone else’s PR with my own commits.
GitHubThursday, November 9th, 2023 at 4:03:16 PM GMT-05:00
UnknownFriday, November 10th, 2023 at 4:23:06 AM GMT-05:00
GitHubFriday, November 10th, 2023 at 4:23:05 AM GMT-05:00
UnknownFriday, November 10th, 2023 at 4:23:06 AM GMT-05:00
GitHubFriday, November 10th, 2023 at 4:35:53 AM GMT-05:00
UnknownFriday, November 10th, 2023 at 4:35:54 AM GMT-05:00
GitHubFriday, November 10th, 2023 at 4:35:54 AM GMT-05:00
UnknownFriday, November 10th, 2023 at 4:35:54 AM GMT-05:00
UnknownFriday, November 10th, 2023 at 4:35:54 AM GMT-05:00
GitHubFriday, November 10th, 2023 at 4:47:06 AM GMT-05:00
UnknownFriday, November 10th, 2023 at 5:11:07 AM GMT-05:00
GitHubFriday, November 10th, 2023 at 5:11:07 AM GMT-05:00
UnknownFriday, November 10th, 2023 at 5:11:07 AM GMT-05:00
GitHubFriday, November 10th, 2023 at 7:19:41 AM GMT-05:00
UnknownFriday, November 10th, 2023 at 7:19:42 AM GMT-05:00
GitHubFriday, November 10th, 2023 at 7:20:07 AM GMT-05:00
UnknownMonday, November 13th, 2023 at 11:28:57 AM GMT-05:00
GitHubFriday, November 10th, 2023 at 10:11:13 AM GMT-05:00
UnknownFriday, November 10th, 2023 at 10:16:05 AM GMT-05:00
GitHubFriday, November 10th, 2023 at 10:16:05 AM GMT-05:00
UnknownFriday, November 10th, 2023 at 10:16:05 AM GMT-05:00
GitHubFriday, November 10th, 2023 at 4:01:19 PM GMT-05:00
GitHubSaturday, November 11th, 2023 at 11:28:57 PM GMT-05:00
UnknownMonday, November 13th, 2023 at 1:41:45 AM GMT-05:00
GitHubSaturday, November 11th, 2023 at 11:29:08 PM GMT-05:00
UnknownMonday, November 13th, 2023 at 1:41:28 AM GMT-05:00
GitHubSaturday, November 11th, 2023 at 11:29:33 PM GMT-05:00
UnknownMonday, November 13th, 2023 at 1:42:41 AM GMT-05:00
GitHubSaturday, November 11th, 2023 at 11:30:01 PM GMT-05:00
UnknownMonday, November 13th, 2023 at 5:26:08 AM GMT-05:00
GitHubSaturday, November 11th, 2023 at 11:30:02 PM GMT-05:00
UnknownMonday, November 13th, 2023 at 1:44:44 AM GMT-05:00
GitHubSaturday, November 11th, 2023 at 11:39:08 PM GMT-05:00
UnknownMonday, November 13th, 2023 at 1:45:27 AM GMT-05:00
GitHubSaturday, November 11th, 2023 at 11:39:22 PM GMT-05:00
UnknownMonday, November 13th, 2023 at 1:44:08 AM GMT-05:00
GitHubSaturday, November 11th, 2023 at 11:39:25 PM GMT-05:00
UnknownMonday, November 13th, 2023 at 1:40:43 AM GMT-05:00
GitHubSaturday, November 11th, 2023 at 11:39:42 PM GMT-05:00
UnknownMonday, November 13th, 2023 at 5:25:38 AM GMT-05:00
GitHubSaturday, November 11th, 2023 at 11:39:54 PM GMT-05:00
UnknownMonday, November 13th, 2023 at 11:32:29 AM GMT-05:00
GitHubMonday, November 13th, 2023 at 1:40:43 AM GMT-05:00
UnknownMonday, November 13th, 2023 at 1:40:43 AM GMT-05:00
GitHubMonday, November 13th, 2023 at 1:41:28 AM GMT-05:00
UnknownMonday, November 13th, 2023 at 1:41:28 AM GMT-05:00
GitHubMonday, November 13th, 2023 at 1:41:44 AM GMT-05:00
UnknownMonday, November 13th, 2023 at 1:41:45 AM GMT-05:00
GitHubMonday, November 13th, 2023 at 1:42:40 AM GMT-05:00
UnknownMonday, November 13th, 2023 at 1:42:41 AM GMT-05:00
GitHubMonday, November 13th, 2023 at 1:44:07 AM GMT-05:00
UnknownMonday, November 13th, 2023 at 1:44:08 AM GMT-05:00
GitHubMonday, November 13th, 2023 at 1:44:43 AM GMT-05:00
UnknownMonday, November 13th, 2023 at 1:44:44 AM GMT-05:00
GitHubMonday, November 13th, 2023 at 1:45:27 AM GMT-05:00
UnknownMonday, November 13th, 2023 at 1:45:27 AM GMT-05:00
UnknownMonday, November 13th, 2023 at 4:15:17 AM GMT-05:00
GitHubMonday, November 13th, 2023 at 5:25:38 AM GMT-05:00
UnknownMonday, November 13th, 2023 at 5:25:38 AM GMT-05:00
GitHubMonday, November 13th, 2023 at 5:26:08 AM GMT-05:00
UnknownMonday, November 13th, 2023 at 5:26:08 AM GMT-05:00
GitHubMonday, November 13th, 2023 at 5:52:02 AM GMT-05:00
UnknownMonday, November 13th, 2023 at 5:52:03 AM GMT-05:00
UnknownMonday, November 13th, 2023 at 5:52:03 AM GMT-05:00
GitHubMonday, November 13th, 2023 at 10:44:42 AM GMT-05:00
GitHubMonday, November 13th, 2023 at 11:28:56 AM GMT-05:00
UnknownMonday, November 13th, 2023 at 11:28:57 AM GMT-05:00
GitHubMonday, November 13th, 2023 at 11:29:21 AM GMT-05:00
UnknownMonday, November 13th, 2023 at 11:31:14 AM GMT-05:00
GitHubMonday, November 13th, 2023 at 11:29:22 AM GMT-05:00
UnknownMonday, November 13th, 2023 at 11:31:43 AM GMT-05:00
GitHubMonday, November 13th, 2023 at 11:29:27 AM GMT-05:00
UnknownMonday, November 13th, 2023 at 11:31:33 AM GMT-05:00
GitHubMonday, November 13th, 2023 at 11:30:11 AM GMT-05:00
UnknownMonday, November 13th, 2023 at 11:31:59 AM GMT-05:00
GitHubMonday, November 13th, 2023 at 11:31:14 AM GMT-05:00
UnknownMonday, November 13th, 2023 at 11:31:14 AM GMT-05:00
GitHubMonday, November 13th, 2023 at 11:31:33 AM GMT-05:00
UnknownMonday, November 13th, 2023 at 11:31:33 AM GMT-05:00
GitHubMonday, November 13th, 2023 at 11:31:43 AM GMT-05:00
UnknownMonday, November 13th, 2023 at 11:31:43 AM GMT-05:00
GitHubMonday, November 13th, 2023 at 11:31:58 AM GMT-05:00
UnknownMonday, November 13th, 2023 at 11:31:59 AM GMT-05:00
GitHubMonday, November 13th, 2023 at 11:32:29 AM GMT-05:00
UnknownMonday, November 13th, 2023 at 11:32:29 AM GMT-05:00
GitHubTuesday, November 14th, 2023 at 11:14:01 AM GMT-05:00
UnknownThursday, November 16th, 2023 at 9:23:57 AM GMT-05:00
GitHubTuesday, November 14th, 2023 at 6:58:02 PM GMT-05:00
UnknownWednesday, November 15th, 2023 at 9:23:51 AM GMT-05:00
GitHubTuesday, November 14th, 2023 at 7:45:38 PM GMT-05:00
UnknownTuesday, November 14th, 2023 at 7:45:39 PM GMT-05:00
UnknownThursday, November 16th, 2023 at 4:31:48 PM GMT-05:00
UnknownTuesday, November 14th, 2023 at 7:45:39 PM GMT-05:00
GitHubTuesday, November 14th, 2023 at 9:11:33 PM GMT-05:00
GitHubWednesday, November 15th, 2023 at 4:33:43 AM GMT-05:00
UnknownWednesday, November 15th, 2023 at 5:07:16 AM GMT-05:00
UnknownWednesday, November 15th, 2023 at 4:56:42 AM GMT-05:00
UnknownWednesday, November 15th, 2023 at 4:58:47 AM GMT-05:00
GitHubWednesday, November 15th, 2023 at 5:00:01 AM GMT-05:00
UnknownWednesday, November 15th, 2023 at 5:00:01 AM GMT-05:00
UnknownWednesday, November 15th, 2023 at 5:00:01 AM GMT-05:00
GitHubWednesday, November 15th, 2023 at 5:00:32 AM GMT-05:00
UnknownWednesday, November 15th, 2023 at 5:00:32 AM GMT-05:00
UnknownWednesday, November 15th, 2023 at 5:00:32 AM GMT-05:00
GitHubWednesday, November 15th, 2023 at 5:00:50 AM GMT-05:00
UnknownWednesday, November 15th, 2023 at 5:00:51 AM GMT-05:00
UnknownWednesday, November 15th, 2023 at 5:00:51 AM GMT-05:00
GitHubWednesday, November 15th, 2023 at 5:07:16 AM GMT-05:00
UnknownWednesday, November 15th, 2023 at 5:07:16 AM GMT-05:00
GitHubWednesday, November 15th, 2023 at 9:23:51 AM GMT-05:00
UnknownWednesday, November 15th, 2023 at 9:23:51 AM GMT-05:00
GitHubThursday, November 16th, 2023 at 9:23:56 AM GMT-05:00
UnknownThursday, November 16th, 2023 at 9:23:57 AM GMT-05:00
Harel SheinThursday, November 16th, 2023 at 11:40:06 AM GMT-05:00
@Harel Shein has joined the channel
GitHubThursday, November 16th, 2023 at 4:31:47 PM GMT-05:00
UnknownThursday, November 16th, 2023 at 4:31:48 PM GMT-05:00
GitHubFriday, November 17th, 2023 at 8:51:37 AM GMT-05:00
GitHubFriday, November 17th, 2023 at 3:11:03 PM GMT-05:00
UnknownFriday, November 17th, 2023 at 4:46:07 PM GMT-05:00
UnknownFriday, November 17th, 2023 at 4:46:07 PM GMT-05:00
GitHubFriday, November 17th, 2023 at 4:51:57 PM GMT-05:00
\ No newline at end of file diff --git a/html/avatars/U01UYQB6JQH.png b/html/avatars/U01UYQB6JQH.png deleted file mode 100644 index 53b53ee..0000000 Binary files a/html/avatars/U01UYQB6JQH.png and /dev/null differ diff --git a/html/avatars/U05P1B3476H.png b/html/avatars/U05P1B3476H.png deleted file mode 100644 index a422ec6..0000000 Binary files a/html/avatars/U05P1B3476H.png and /dev/null differ diff --git a/html/avatars/U06204TBQQN.jpg b/html/avatars/U06204TBQQN.jpg deleted file mode 100644 index db2d3d4..0000000 Binary files a/html/avatars/U06204TBQQN.jpg and /dev/null differ diff --git a/html/files/.DS_Store b/html/files/.DS_Store deleted file mode 100644 index 3f7a542..0000000 Binary files a/html/files/.DS_Store and /dev/null differ diff --git a/index.html b/index.html deleted file mode 100644 index 8c6897b..0000000 --- a/index.html +++ /dev/null @@ -1,10 +0,0 @@ -Slack

Public Channels

Private Channels

DMs

Group DMs

Bots

Archived Public Channels

Archived Private Channels

DMs (Deleted Users)

\ No newline at end of file diff --git a/slack-archive/.last-successful-run b/slack-archive/.last-successful-run new file mode 100644 index 0000000..b4ae3a1 --- /dev/null +++ b/slack-archive/.last-successful-run @@ -0,0 +1 @@ +2023-11-21T14:26:35.411Z \ No newline at end of file diff --git a/slack-archive/data/C01CK9T7HKR.json b/slack-archive/data/C01CK9T7HKR.json new file mode 100644 index 0000000..1b23810 --- /dev/null +++ b/slack-archive/data/C01CK9T7HKR.json @@ -0,0 +1,32963 @@ +[ + { + "client_msg_id": "c8703ac5-44a3-41fb-9fe7-b2ada2736ef9", + "type": "message", + "text": "Hi Everyone, first of all - big shout to all contributors - You do amazing job here.\nI want to use OpenLineage in our project - to do so I want to setup some POC and experiment with possibilities library provides - I start working on sample from the conference talk: but when I go into spark transformation after staring context with openlineage I have issues with _SessionHiveMetaStoreClient on section 3_- does anyone has other plain sample to play with, to not setup everything from scratch?", + "user": "U066S97A90C", + "ts": "1700568128.192669", + "blocks": [ + { + "type": "rich_text", + "block_id": "LZMJn", + "elements": [ + { + "type": "rich_text_section", + "elements": [ + { + "type": "text", + "text": "Hi Everyone, first of all - big shout to all contributors - You do amazing job here.\nI want to use OpenLineage in our project - to do so I want to setup some POC and experiment with possibilities library provides - I start working on sample from the conference talk: " + }, + { + "type": "link", + "url": "https://github.com/getindata/openlineage-bbuzz2023-column-lineage" + }, + { + "type": "text", + "text": " but when I go into spark transformation after staring context with openlineage I have issues with " + }, + { + "type": "text", + "text": "SessionHiveMetaStoreClient on section 3", + "style": { + "italic": true + } + }, + { + "type": "text", + "text": "- does anyone has other plain sample to play with, to not setup everything from scratch?" + } + ] + } + ] + } + ], + "team": "T01CWUYP5AR", + "attachments": [ + { + "id": 1, + "color": "24292f", + "bot_id": "B01VA0FB340", + "app_unfurl_url": "https://github.com/getindata/openlineage-bbuzz2023-column-lineage", + "is_app_unfurl": true, + "app_id": "A01BP7R4KNY", + "fallback": "getindata/openlineage-bbuzz2023-column-lineage", + "title": "getindata/openlineage-bbuzz2023-column-lineage", + "fields": [ + { + "value": "Jupyter Notebook", + "title": "Language", + "short": true + }, + { + "value": "5 months ago", + "title": "Last updated", + "short": true + } + ] + } + ], + "thread_ts": "1700568128.192669", + "reply_count": 9, + "reply_users_count": 3, + "latest_reply": "1700576519.927839", + "reply_users": [ + "U01RA9B5GG2", + "U066S97A90C", + "U02MK6YNAQ5" + ], + "is_locked": false, + "subscribed": false, + "replies": [ + { + "client_msg_id": "78925e07-2703-4759-90dd-9202405c52cf", + "type": "message", + "text": "Can you provide details about those issues? Like exceptions, logs, details of the jobs and how do you run them?", + "user": "U01RA9B5GG2", + "ts": "1700570220.696829", + "blocks": [ + { + "type": "rich_text", + "block_id": "Cvx2D", + "elements": [ + { + "type": "rich_text_section", + "elements": [ + { + "type": "text", + "text": "Can you provide details about those issues? Like exceptions, logs, details of the jobs and how do you run them?" + } + ] + } + ] + } + ], + "team": "T01CWUYP5AR", + "thread_ts": "1700568128.192669", + "parent_user_id": "U066S97A90C" + }, + { + "client_msg_id": "59b35b8a-9b98-4517-aa83-de41a9855e29", + "type": "message", + "text": "Hi <@U01RA9B5GG2> - I rerun docker container after deleting metadata_db folder possibly created by other local test, and fix this one but got problem with OpenLineageListener - during initialization of spark:\nwhile I execute:\n```spark = (SparkSession.builder.master('local')\n .appName('Food Delivery')\n .config('spark.extraListeners', 'io.openlineage.spark.agent.OpenLineageSparkListener')\n .config('spark.jars', '<local-path>/openlineage-spark-0.27.2.jar,<local-path>/postgresql-42.6.0.jar')\n .config('spark.openlineage.transport.type', 'http')\n .config('spark.openlineage.transport.url', '')\n .config('spark.openlineage.facets.disabled', '[spark_unknown;spark.logicalPlan]')\n .config('spark.openlineage.namespace', 'food-delivery')\n .config('spark.sql.warehouse.dir', '/tmp/spark-warehouse/')\n .config(\"spark.sql.repl.eagerEval.enabled\", True)\n .enableHiveSupport()\n .getOrCreate())```\nI got\n```Py4JJavaError: An error occurred while calling None.org.apache.spark.api.java.JavaSparkContext.\n: org.apache.spark.SparkException: Exception when registering SparkListener\n\tat org.apache.spark.SparkContext.setupAndStartListenerBus(SparkContext.scala:2563)\n\tat org.apache.spark.SparkContext.<init>(SparkContext.scala:643)\n\tat org.apache.spark.api.java.JavaSparkContext.<init>(JavaSparkContext.scala:58)\n\tat java.base/jdk.internal.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)\n\tat java.base/jdk.internal.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:77)\n\tat java.base/jdk.internal.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)\n\tat java.base/java.lang.reflect.Constructor.newInstanceWithCaller(Constructor.java:499)\n\tat java.base/java.lang.reflect.Constructor.newInstance(Constructor.java:480)\n\tat py4j.reflection.MethodInvoker.invoke(MethodInvoker.java:247)\n\tat py4j.reflection.ReflectionEngine.invoke(ReflectionEngine.java:357)\n\tat py4j.Gateway.invoke(Gateway.java:238)\n\tat py4j.commands.ConstructorCommand.invokeConstructor(ConstructorCommand.java:80)\n\tat py4j.commands.ConstructorCommand.execute(ConstructorCommand.java:69)\n\tat py4j.ClientServerConnection.waitForCommands(ClientServerConnection.java:182)\n\tat py4j.ClientServerConnection.run(ClientServerConnection.java:106)\n\tat java.base/java.lang.Thread.run(Thread.java:833)\nCaused by: java.lang.ClassNotFoundException: io.openlineage.spark.agent.OpenLineageSparkListener\n\tat java.base/java.net.URLClassLoader.findClass(URLClassLoader.java:445)\n\tat java.base/java.lang.ClassLoader.loadClass(ClassLoader.java:587)\n\tat java.base/java.lang.ClassLoader.loadClass(ClassLoader.java:520)\n\tat java.base/java.lang.Class.forName0(Native Method)\n\tat java.base/java.lang.Class.forName(Class.java:467)\n\tat org.apache.spark.util.Utils$.classForName(Utils.scala:218)\n\tat org.apache.spark.util.Utils$.$anonfun$loadExtensions$1(Utils.scala:2921)\n\tat scala.collection.TraversableLike.$anonfun$flatMap$1(TraversableLike.scala:293)\n\tat scala.collection.mutable.ResizableArray.foreach(ResizableArray.scala:62)\n\tat scala.collection.mutable.ResizableArray.foreach$(ResizableArray.scala:55)\n\tat scala.collection.mutable.ArrayBuffer.foreach(ArrayBuffer.scala:49)\n\tat scala.collection.TraversableLike.flatMap(TraversableLike.scala:293)\n\tat scala.collection.TraversableLike.flatMap$(TraversableLike.scala:290)\n\tat scala.collection.AbstractTraversable.flatMap(Traversable.scala:108)\n\tat org.apache.spark.util.Utils$.loadExtensions(Utils.scala:2919)\n\tat org.apache.spark.SparkContext.$anonfun$setupAndStartListenerBus$1(SparkContext.scala:2552)\n\tat org.apache.spark.SparkContext.$anonfun$setupAndStartListenerBus$1$adapted(SparkContext.scala:2551)\n\tat scala.Option.foreach(Option.scala:407)\n\tat org.apache.spark.SparkContext.setupAndStartListenerBus(SparkContext.scala:2551)\n\t... 15 more```\nlooks like by some reasons jars are not loaded - need to look into it", + "user": "U066S97A90C", + "ts": "1700570737.279069", + "blocks": [ + { + "type": "rich_text", + "block_id": "modz/", + "elements": [ + { + "type": "rich_text_section", + "elements": [ + { + "type": "text", + "text": "Hi " + }, + { + "type": "user", + "user_id": "U01RA9B5GG2" + }, + { + "type": "text", + "text": " - I rerun docker container after deleting metadata_db folder possibly created by other local test, and fix this one but got problem with OpenLineageListener - during initialization of spark:\nwhile I execute:\n" + } + ] + }, + { + "type": "rich_text_preformatted", + "elements": [ + { + "type": "text", + "text": "spark = (SparkSession.builder.master('local')\n .appName('Food Delivery')\n .config('spark.extraListeners', 'io.openlineage.spark.agent.OpenLineageSparkListener')\n .config('spark.jars', '/openlineage-spark-0.27.2.jar,/postgresql-42.6.0.jar')\n .config('spark.openlineage.transport.type', 'http')\n .config('spark.openlineage.transport.url', '" + }, + { + "type": "link", + "url": "http://api:5000" + }, + { + "type": "text", + "text": "')\n .config('spark.openlineage.facets.disabled', '[spark_unknown;spark.logicalPlan]')\n .config('spark.openlineage.namespace', 'food-delivery')\n .config('spark.sql.warehouse.dir', '/tmp/spark-warehouse/')\n .config(\"spark.sql.repl.eagerEval.enabled\", True)\n .enableHiveSupport()\n .getOrCreate())" + } + ], + "border": 0 + }, + { + "type": "rich_text_section", + "elements": [ + { + "type": "text", + "text": "I got\n" + } + ] + }, + { + "type": "rich_text_preformatted", + "elements": [ + { + "type": "text", + "text": "Py4JJavaError: An error occurred while calling None.org.apache.spark.api.java.JavaSparkContext.\n: org.apache.spark.SparkException: Exception when registering SparkListener\n\tat org.apache.spark.SparkContext.setupAndStartListenerBus(SparkContext.scala:2563)\n\tat org.apache.spark.SparkContext.(SparkContext.scala:643)\n\tat org.apache.spark.api.java.JavaSparkContext.(JavaSparkContext.scala:58)\n\tat java.base/jdk.internal.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)\n\tat java.base/jdk.internal.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:77)\n\tat java.base/jdk.internal.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)\n\tat java.base/java.lang.reflect.Constructor.newInstanceWithCaller(Constructor.java:499)\n\tat java.base/java.lang.reflect.Constructor.newInstance(Constructor.java:480)\n\tat py4j.reflection.MethodInvoker.invoke(MethodInvoker.java:247)\n\tat py4j.reflection.ReflectionEngine.invoke(ReflectionEngine.java:357)\n\tat py4j.Gateway.invoke(Gateway.java:238)\n\tat py4j.commands.ConstructorCommand.invokeConstructor(ConstructorCommand.java:80)\n\tat py4j.commands.ConstructorCommand.execute(ConstructorCommand.java:69)\n\tat py4j.ClientServerConnection.waitForCommands(ClientServerConnection.java:182)\n\tat py4j.ClientServerConnection.run(ClientServerConnection.java:106)\n\tat java.base/java.lang.Thread.run(Thread.java:833)\nCaused by: java.lang.ClassNotFoundException: io.openlineage.spark.agent.OpenLineageSparkListener\n\tat java.base/java.net.URLClassLoader.findClass(URLClassLoader.java:445)\n\tat java.base/java.lang.ClassLoader.loadClass(ClassLoader.java:587)\n\tat java.base/java.lang.ClassLoader.loadClass(ClassLoader.java:520)\n\tat java.base/java.lang.Class.forName0(Native Method)\n\tat java.base/java.lang.Class.forName(Class.java:467)\n\tat org.apache.spark.util.Utils$.classForName(Utils.scala:218)\n\tat org.apache.spark.util.Utils$.$anonfun$loadExtensions$1(Utils.scala:2921)\n\tat scala.collection.TraversableLike.$anonfun$flatMap$1(TraversableLike.scala:293)\n\tat scala.collection.mutable.ResizableArray.foreach(ResizableArray.scala:62)\n\tat scala.collection.mutable.ResizableArray.foreach$(ResizableArray.scala:55)\n\tat scala.collection.mutable.ArrayBuffer.foreach(ArrayBuffer.scala:49)\n\tat scala.collection.TraversableLike.flatMap(TraversableLike.scala:293)\n\tat scala.collection.TraversableLike.flatMap$(TraversableLike.scala:290)\n\tat scala.collection.AbstractTraversable.flatMap(Traversable.scala:108)\n\tat org.apache.spark.util.Utils$.loadExtensions(Utils.scala:2919)\n\tat org.apache.spark.SparkContext.$anonfun$setupAndStartListenerBus$1(SparkContext.scala:2552)\n\tat org.apache.spark.SparkContext.$anonfun$setupAndStartListenerBus$1$adapted(SparkContext.scala:2551)\n\tat scala.Option.foreach(Option.scala:407)\n\tat org.apache.spark.SparkContext.setupAndStartListenerBus(SparkContext.scala:2551)\n\t... 15 more" + } + ], + "border": 0 + }, + { + "type": "rich_text_section", + "elements": [ + { + "type": "text", + "text": "looks like by some reasons jars are not loaded - need to look into it" + } + ] + } + ] + } + ], + "team": "T01CWUYP5AR", + "thread_ts": "1700568128.192669", + "parent_user_id": "U066S97A90C" + }, + { + "client_msg_id": "c75639c8-300b-41e0-8548-bc6a0bf8fcd5", + "type": "message", + "text": ":thinking_face: Jars are added during image building: ", + "user": "U01RA9B5GG2", + "ts": "1700571489.504839", + "blocks": [ + { + "type": "rich_text", + "block_id": "ek23V", + "elements": [ + { + "type": "rich_text_section", + "elements": [ + { + "type": "emoji", + "name": "thinking_face", + "unicode": "1f914" + }, + { + "type": "text", + "text": " Jars are added during image building: " + }, + { + "type": "link", + "url": "https://github.com/getindata/openlineage-bbuzz2023-column-lineage/blob/main/Dockerfile#L12C1-L12C29" + } + ] + } + ] + } + ], + "team": "T01CWUYP5AR", + "thread_ts": "1700568128.192669", + "parent_user_id": "U066S97A90C" + }, + { + "client_msg_id": "eb7d3d59-bb15-4c16-a7fc-872af3828c7e", + "type": "message", + "text": "are you sure `<local-path>` is right?", + "user": "U01RA9B5GG2", + "ts": "1700571508.332809", + "blocks": [ + { + "type": "rich_text", + "block_id": "6yFIl", + "elements": [ + { + "type": "rich_text_section", + "elements": [ + { + "type": "text", + "text": "are you sure " + }, + { + "type": "text", + "text": "", + "style": { + "code": true + } + }, + { + "type": "text", + "text": " is right?" + } + ] + } + ] + } + ], + "team": "T01CWUYP5AR", + "thread_ts": "1700568128.192669", + "parent_user_id": "U066S97A90C" + }, + { + "client_msg_id": "afa11cb3-a6d6-4766-aaf1-37d25390cab2", + "type": "message", + "text": "yes, it's same as in sample - wondering why it's not get added:\n```from pyspark.sql import SparkSession\n\nspark = (SparkSession.builder.master('local')\n .appName('Food Delivery')\n .config('spark.jars', '/home/jovyan/jars/openlineage-spark-0.27.2.jar,/home/jovyan/jars/postgresql-42.6.0.jar')\n .config('spark.sql.warehouse.dir', '/tmp/spark-warehouse/')\n .config(\"spark.sql.repl.eagerEval.enabled\", True)\n .enableHiveSupport()\n .getOrCreate())\n\nprint(().listJars())\n\nVector()```", + "user": "U066S97A90C", + "ts": "1700571649.351379", + "blocks": [ + { + "type": "rich_text", + "block_id": "Pjc3w", + "elements": [ + { + "type": "rich_text_section", + "elements": [ + { + "type": "text", + "text": "yes, it's same as in sample - wondering why it's not get added:\n" + } + ] + }, + { + "type": "rich_text_preformatted", + "elements": [ + { + "type": "text", + "text": "from pyspark.sql import SparkSession\n\nspark = (SparkSession.builder.master('local')\n .appName('Food Delivery')\n .config('spark.jars', '/home/jovyan/jars/openlineage-spark-0.27.2.jar,/home/jovyan/jars/postgresql-42.6.0.jar')\n .config('spark.sql.warehouse.dir', '/tmp/spark-warehouse/')\n .config(\"spark.sql.repl.eagerEval.enabled\", True)\n .enableHiveSupport()\n .getOrCreate())\n\nprint(spark.sparkContext._jsc.sc().listJars())\n\nVector()" + } + ], + "border": 0 + } + ] + } + ], + "team": "T01CWUYP5AR", + "thread_ts": "1700568128.192669", + "parent_user_id": "U066S97A90C" + }, + { + "client_msg_id": "5a0d71ab-9451-4418-bfd1-ee136218e71d", + "type": "message", + "text": "can you make sure jars are in this directory? just by `docker run --entrypoint /usr/local/bin/bash IMAGE_NAME \"ls /home/jovyan/jars\"`", + "user": "U01RA9B5GG2", + "ts": "1700571871.112339", + "blocks": [ + { + "type": "rich_text", + "block_id": "RLxB1", + "elements": [ + { + "type": "rich_text_section", + "elements": [ + { + "type": "text", + "text": "can you make sure jars are in this directory? just by " + }, + { + "type": "text", + "text": "docker run --entrypoint /usr/local/bin/bash IMAGE_NAME \"ls /home/jovyan/jars\"", + "style": { + "code": true + } + } + ] + } + ] + } + ], + "team": "T01CWUYP5AR", + "thread_ts": "1700568128.192669", + "parent_user_id": "U066S97A90C" + }, + { + "client_msg_id": "913d9441-d9ac-4616-8eda-3d38d643e491", + "type": "message", + "text": "another option to try is to replace `spark.jars` with `spark.jars.packages` `io.openlineage:openlineage-spark:1.5.0,org.postgresql:postgresql:42.7.0`", + "user": "U01RA9B5GG2", + "ts": "1700571987.624519", + "blocks": [ + { + "type": "rich_text", + "block_id": "hfgFo", + "elements": [ + { + "type": "rich_text_section", + "elements": [ + { + "type": "text", + "text": "another option to try is to replace " + }, + { + "type": "text", + "text": "spark.jars", + "style": { + "code": true + } + }, + { + "type": "text", + "text": " with " + }, + { + "type": "text", + "text": "spark.jars.packages", + "style": { + "code": true + } + }, + { + "type": "text", + "text": " " + }, + { + "type": "text", + "text": "io.openlineage:openlineage-spark:1.5.0,org.postgresql:postgresql:42.7.0", + "style": { + "code": true + } + } + ] + } + ] + } + ], + "team": "T01CWUYP5AR", + "thread_ts": "1700568128.192669", + "parent_user_id": "U066S97A90C" + }, + { + "client_msg_id": "bdfc78ad-1de4-4ecc-b731-91b2d2c7d9fd", + "type": "message", + "text": "I think this was done for the purpose of presentation to make sure the demo will work without internet access. This can be the reason to add jar manually to a docker. `openlineage-spark` can be added to Spark via `spark.jar.packages` , like we do here ", + "user": "U02MK6YNAQ5", + "ts": "1700572614.636549", + "blocks": [ + { + "type": "rich_text", + "block_id": "4fM4X", + "elements": [ + { + "type": "rich_text_section", + "elements": [ + { + "type": "text", + "text": "I think this was done for the purpose of presentation to make sure the demo will work without internet access. This can be the reason to add jar manually to a docker. " + }, + { + "type": "text", + "text": "openlineage-spark", + "style": { + "code": true + } + }, + { + "type": "text", + "text": " can be added to Spark via " + }, + { + "type": "text", + "text": "spark.jar.packages", + "style": { + "code": true + } + }, + { + "type": "text", + "text": " , like we do here " + }, + { + "type": "link", + "url": "https://openlineage.io/docs/integrations/spark/quickstart_local" + } + ] + } + ] + } + ], + "team": "T01CWUYP5AR", + "attachments": [ + { + "from_url": "https://openlineage.io/docs/integrations/spark/quickstart_local", + "service_icon": "https://openlineage.io/img/favicon.ico", + "id": 1, + "original_url": "https://openlineage.io/docs/integrations/spark/quickstart_local", + "fallback": "Quickstart with Jupyter | OpenLineage", + "text": "Trying out the Spark integration is super easy if you already have Docker Desktop and git installed.", + "title": "Quickstart with Jupyter | OpenLineage", + "title_link": "https://openlineage.io/docs/integrations/spark/quickstart_local", + "service_name": "openlineage.io" + } + ], + "thread_ts": "1700568128.192669", + "parent_user_id": "U066S97A90C" + }, + { + "client_msg_id": "b80b16bc-2f21-4f43-b205-a057ddaa19fe", + "type": "message", + "text": "got it guys - thanks a lot for help - it turns out that spark context from notebook 2 and 3 has come kind of metadata conflict - when I combine those 2 and recreate image to clean up old metadata it works.\nOne more note is that sometimes kernels return weird results but it may be caused by some local nuances - anyways thx !", + "user": "U066S97A90C", + "ts": "1700576519.927839", + "blocks": [ + { + "type": "rich_text", + "block_id": "vjzch", + "elements": [ + { + "type": "rich_text_section", + "elements": [ + { + "type": "text", + "text": "got it guys - thanks a lot for help - it turns out that spark context from notebook 2 and 3 has come kind of metadata conflict - when I combine those 2 and recreate image to clean up old metadata it works.\nOne more note is that sometimes kernels return weird results but it may be caused by some local nuances - anyways thx !" + } + ] + } + ] + } + ], + "team": "T01CWUYP5AR", + "thread_ts": "1700568128.192669", + "parent_user_id": "U066S97A90C" + } + ] + }, + { + "client_msg_id": "25c27b97-4479-4f92-aaaa-bed3ba0eba8c", + "type": "message", + "text": "Hi So we can use openlineage to identify column level lineage with Airflow , Spark? will it also allow to connect to Power BI and derive the downstream column lineage ?", + "user": "U066CNW85D3", + "ts": "1700456258.614309", + "blocks": [ + { + "type": "rich_text", + "block_id": "maShE", + "elements": [ + { + "type": "rich_text_section", + "elements": [ + { + "type": "text", + "text": "Hi So we can use openlineage to identify column level lineage with Airflow , Spark? will it also allow to connect to Power BI and derive the downstream column lineage ?" + } + ] + } + ] + } + ], + "team": "T01CWUYP5AR", + "thread_ts": "1700456258.614309", + "reply_count": 3, + "reply_users_count": 2, + "latest_reply": "1700571550.031889", + "reply_users": [ + "U01RA9B5GG2", + "U066CNW85D3" + ], + "is_locked": false, + "subscribed": false, + "replies": [ + { + "client_msg_id": "f6f8055d-1939-45f1-a446-43f5652da88b", + "type": "message", + "text": "Yes, it works with Airflow and Spark - there is caveat that amount of operators that support it on Airflow side is fairly small and limited generally to most popular SQL operators.\n> will it also allow to connect to Power BI and derive the downstream column lineage ?\nNo, there is no such feature _yet_ :slightly_smiling_face:\nHowever, there's nothing preventing this - if you wish to work on such implementation, we'd be happy to help.", + "user": "U01RA9B5GG2", + "ts": "1700478456.679879", + "blocks": [ + { + "type": "rich_text", + "block_id": "W+RUb", + "elements": [ + { + "type": "rich_text_section", + "elements": [ + { + "type": "text", + "text": "Yes, it works with Airflow and Spark - there is caveat that amount of operators that support it on Airflow side is fairly small and limited generally to most popular SQL operators.\n" + } + ] + }, + { + "type": "rich_text_quote", + "elements": [ + { + "type": "text", + "text": "will it also allow to connect to Power BI and derive the downstream column lineage ?" + } + ] + }, + { + "type": "rich_text_section", + "elements": [ + { + "type": "text", + "text": "No, there is no such feature " + }, + { + "type": "text", + "text": "yet ", + "style": { + "italic": true + } + }, + { + "type": "emoji", + "name": "slightly_smiling_face", + "unicode": "1f642" + }, + { + "type": "text", + "text": "\nHowever, there's nothing preventing this - if you wish to work on such implementation, we'd be happy to help." + } + ] + } + ] + } + ], + "team": "T01CWUYP5AR", + "thread_ts": "1700456258.614309", + "parent_user_id": "U066CNW85D3" + }, + { + "client_msg_id": "3b2d2349-9b67-4745-a2f2-217d82315108", + "type": "message", + "text": "Thank you Maciej Obuchowski for the update. Currently we are looking out for a tool which can support connecting to Power Bi and pull column level lineage information for reports and dashboards. How this can be achieved with OL ? Can you give some idea?", + "user": "U066CNW85D3", + "ts": "1700544011.513999", + "blocks": [ + { + "type": "rich_text", + "block_id": "+u0dj", + "elements": [ + { + "type": "rich_text_section", + "elements": [ + { + "type": "text", + "text": "Thank you Maciej Obuchowski for the update. Currently we are looking out for a tool which can support connecting to Power Bi and pull column level lineage information for reports and dashboards. How this can be achieved with OL ? Can you give some idea?" + } + ] + } + ] + } + ], + "team": "T01CWUYP5AR", + "edited": { + "user": "U066CNW85D3", + "ts": "1700544159.000000" + }, + "thread_ts": "1700456258.614309", + "parent_user_id": "U066CNW85D3" + }, + { + "client_msg_id": "3c666be0-c6ff-48f8-bc2c-af26eb4dfb0e", + "type": "message", + "text": "I don't think I can help you with that now, unless you want to work on your own integration with PowerBI :slightly_frowning_face:", + "user": "U01RA9B5GG2", + "ts": "1700571550.031889", + "blocks": [ + { + "type": "rich_text", + "block_id": "XtCzB", + "elements": [ + { + "type": "rich_text_section", + "elements": [ + { + "type": "text", + "text": "I don't think I can help you with that now, unless you want to work on your own integration with PowerBI " + }, + { + "type": "emoji", + "name": "slightly_frowning_face", + "unicode": "1f641" + } + ] + } + ] + } + ], + "team": "T01CWUYP5AR", + "thread_ts": "1700456258.614309", + "parent_user_id": "U066CNW85D3" + } + ] + }, + { + "client_msg_id": "aff93a22-bd07-470a-92b8-b4aafd17b114", + "type": "message", + "text": "what are the pros and cons of OL. we often talk about positives to market it but what are the pain points using OL,how it's addressing user issues?", + "user": "U066HKFCHUG", + "ts": "1700064658.956769", + "blocks": [ + { + "type": "rich_text", + "block_id": "af/Ig", + "elements": [ + { + "type": "rich_text_section", + "elements": [ + { + "type": "text", + "text": "what are the pros and cons of OL. we often talk about positives to market it but what are the pain points using OL,how it's addressing user issues?" + } + ] + } + ] + } + ], + "team": "T01CWUYP5AR", + "thread_ts": "1700064658.956769", + "reply_count": 1, + "reply_users_count": 1, + "latest_reply": "1700159922.652539", + "reply_users": [ + "U02LXF3HUN7" + ], + "is_locked": false, + "subscribed": true, + "last_read": "1700159922.652539", + "replies": [ + { + "client_msg_id": "00835fbb-223f-4679-b436-23ffc9aafffe", + "type": "message", + "text": "Hi <@U066HKFCHUG>, thanks for your question. We’ve heard that OpenLineage is attractive because of its desirable integrations, including a best-in-class Spark integration, its extensibility, the fact that it’s not destructive, and the fact that it’s open source. I’m not aware of pain points per se, but there are certainly features and integrations that we wish we could focus on but can’t at the moment — like the Dagster integration, which needs a new maintainer. OpenLineage is like any other open standard in that ecosystem coverage is a constant process rather than a journey, and it requires contributions in order to get close to 100%. Thankfully, we are gaining users and contributors all the time, and integrations are being added or improved upon daily. See the Ecosystem page on the website for a list of consumers and producers and links to more resources, and check out the for the codebase, commit history, contributors, governance procedures, and more. We’re quick to respond to messages here and issues on GitHub — usually within one day.", + "user": "U02LXF3HUN7", + "ts": "1700159922.652539", + "blocks": [ + { + "type": "rich_text", + "block_id": "b4e3K", + "elements": [ + { + "type": "rich_text_section", + "elements": [ + { + "type": "text", + "text": "Hi " + }, + { + "type": "user", + "user_id": "U066HKFCHUG" + }, + { + "type": "text", + "text": ", thanks for your question. We’ve heard that OpenLineage is attractive because of its desirable integrations, including a best-in-class Spark integration, its extensibility, the fact that it’s not destructive, and the fact that it’s open source. I’m not aware of pain points per se, but there are certainly features and integrations that we wish we could focus on but can’t at the moment — like the Dagster integration, which needs a new maintainer. OpenLineage is like any other open standard in that ecosystem coverage is a constant process rather than a journey, and it requires contributions in order to get close to 100%. Thankfully, we are gaining users and contributors all the time, and integrations are being added or improved upon daily. See the Ecosystem page on the website for a list of consumers and producers and links to more resources, and check out the " + }, + { + "type": "link", + "url": "https://github.com/OpenLineage/OpenLineage", + "text": "GitHub repo" + }, + { + "type": "text", + "text": " for the codebase, commit history, contributors, governance procedures, and more. We’re quick to respond to messages here and issues on GitHub — usually within one day." + } + ] + } + ] + } + ], + "team": "T01CWUYP5AR", + "attachments": [ + { + "id": 1, + "color": "24292f", + "bot_id": "B01VA0FB340", + "app_unfurl_url": "https://github.com/OpenLineage/OpenLineage", + "is_app_unfurl": true, + "app_id": "A01BP7R4KNY", + "fallback": "OpenLineage/OpenLineage", + "text": "An Open Standard for lineage metadata collection", + "title": "OpenLineage/OpenLineage", + "fields": [ + { + "value": "", + "title": "Website", + "short": true + }, + { + "value": "1449", + "title": "Stars", + "short": true + } + ] + } + ], + "thread_ts": "1700064658.956769", + "parent_user_id": "U066HKFCHUG" + } + ] + }, + { + "client_msg_id": "f21b5e0b-4e53-407a-9f3d-97c6f3d4a986", + "type": "message", + "text": "Can anyone tell me why OL is better than other competitors if you can provide an analysis that would be great", + "user": "U066HKFCHUG", + "ts": "1700064564.825909", + "blocks": [ + { + "type": "rich_text", + "block_id": "V52kz", + "elements": [ + { + "type": "rich_text_section", + "elements": [ + { + "type": "text", + "text": "Can anyone tell me why OL is better than other competitors if you can provide an analysis that would be great" + } + ] + } + ] + } + ], + "team": "T01CWUYP5AR", + "thread_ts": "1700064564.825909", + "reply_count": 1, + "reply_users_count": 1, + "latest_reply": "1700153176.479259", + "reply_users": [ + "U01HNKK4XAM" + ], + "is_locked": false, + "subscribed": false, + "replies": [ + { + "client_msg_id": "b6825ecd-c475-4cff-a701-842af7c6d68e", + "type": "message", + "text": "Hey <@U066HKFCHUG> can you help me understand what you mean by competitors?\nOL is a specification that can be used to solve various problems, so if you have a clear problem statement, maybe I can help with pros/cons for that problem", + "user": "U01HNKK4XAM", + "ts": "1700153176.479259", + "blocks": [ + { + "type": "rich_text", + "block_id": "7oHLi", + "elements": [ + { + "type": "rich_text_section", + "elements": [ + { + "type": "text", + "text": "Hey " + }, + { + "type": "user", + "user_id": "U066HKFCHUG" + }, + { + "type": "text", + "text": " can you help me understand what you mean by competitors?\nOL is a specification that can be used to solve various problems, so if you have a clear problem statement, maybe I can help with pros/cons for that problem" + } + ] + } + ] + } + ], + "team": "T01CWUYP5AR", + "thread_ts": "1700064564.825909", + "parent_user_id": "U066HKFCHUG" + } + ] + }, + { + "client_msg_id": "f2964054-7781-41c2-85ec-e58345c88058", + "type": "message", + "text": "Hi\nCan anyone point me to the deck on how Airflow can be integrated using Openlineage?", + "user": "U066HKFCHUG", + "ts": "1700050644.509419", + "blocks": [ + { + "type": "rich_text", + "block_id": "WgWd5", + "elements": [ + { + "type": "rich_text_section", + "elements": [ + { + "type": "text", + "text": "Hi\nCan anyone point me to the deck on how Airflow can be integrated using Openlineage?" + } + ] + } + ] + } + ], + "team": "T01CWUYP5AR", + "thread_ts": "1700050644.509419", + "reply_count": 2, + "reply_users_count": 2, + "latest_reply": "1700051275.164259", + "reply_users": [ + "U01RA9B5GG2", + "U066HKFCHUG" + ], + "is_locked": false, + "subscribed": false, + "replies": [ + { + "client_msg_id": "ed2ee679-f2cc-4cdc-bbef-c163d66f2850", + "type": "message", + "text": "\n\n", + "user": "U01RA9B5GG2", + "ts": "1700051254.482469", + "blocks": [ + { + "type": "rich_text", + "block_id": "zzeIE", + "elements": [ + { + "type": "rich_text_section", + "elements": [ + { + "type": "link", + "url": "https://airflow.apache.org/docs/apache-airflow-providers-openlineage/stable/index.html" + }, + { + "type": "text", + "text": "\n\n" + }, + { + "type": "link", + "url": "https://airflow.apache.org/docs/apache-airflow-providers-openlineage/stable/guides/user.html" + } + ] + } + ] + } + ], + "team": "T01CWUYP5AR", + "thread_ts": "1700050644.509419", + "parent_user_id": "U066HKFCHUG" + }, + { + "client_msg_id": "b63627c5-ff37-4c5c-9ff0-1a909e3048f4", + "type": "message", + "text": "thank you <@U01RA9B5GG2>", + "user": "U066HKFCHUG", + "ts": "1700051275.164259", + "blocks": [ + { + "type": "rich_text", + "block_id": "GzOOw", + "elements": [ + { + "type": "rich_text_section", + "elements": [ + { + "type": "text", + "text": "thank you " + }, + { + "type": "user", + "user_id": "U01RA9B5GG2" + } + ] + } + ] + } + ], + "team": "T01CWUYP5AR", + "thread_ts": "1700050644.509419", + "parent_user_id": "U066HKFCHUG" + } + ] + }, + { + "client_msg_id": "efc4a680-9991-48df-9816-fe75bb00d4bb", + "type": "message", + "text": "<@U02MK6YNAQ5> I diff CreateReplaceDatasetBuilder.java and CreateReplaceOutputDatasetBuilder.java and they are the same except for the class name, so I am not sure what is causing the change. I also realize you don't have a test case for ADLS", + "user": "U05T8BJD4DU", + "ts": "1699863517.394909", + "blocks": [ + { + "type": "rich_text", + "block_id": "ejWHB", + "elements": [ + { + "type": "rich_text_section", + "elements": [ + { + "type": "user", + "user_id": "U02MK6YNAQ5" + }, + { + "type": "text", + "text": " I diff CreateReplaceDatasetBuilder.java and CreateReplaceOutputDatasetBuilder.java and they are the same except for the class name, so I am not sure what is causing the change. I also realize you don't have a test case for ADLS" + } + ] + } + ] + } + ], + "team": "T01CWUYP5AR", + "thread_ts": "1699863517.394909", + "reply_count": 2, + "reply_users_count": 2, + "latest_reply": "1699916804.305259", + "reply_users": [ + "U02MK6YNAQ5", + "U05T8BJD4DU" + ], + "is_locked": false, + "subscribed": false, + "replies": [ + { + "client_msg_id": "14a63719-a5a6-4cf4-aba3-9fcc8f228db4", + "type": "message", + "text": "Thanks <@U05T8BJD4DU> for your engagement in finding the cause and solution to this issue.\n\nAmong the technical problems, another problem here is that our databricks integration tests are run on AWS and the issue you describe occurs in Azure. I would consider this a primary issue as it is difficult for me to verify the behaviour you describe and fix it with a failing integration test at the start.\n\nAre you able to reproduce the issue on AWS Databricks environment so that we could include it in our integration tests and make sure the behvaiour will not change later on in future?", + "user": "U02MK6YNAQ5", + "ts": "1699869127.461919", + "blocks": [ + { + "type": "rich_text", + "block_id": "LrFjw", + "elements": [ + { + "type": "rich_text_section", + "elements": [ + { + "type": "text", + "text": "Thanks " + }, + { + "type": "user", + "user_id": "U05T8BJD4DU" + }, + { + "type": "text", + "text": " for your engagement in finding the cause and solution to this issue.\n\nAmong the technical problems, another problem here is that our databricks integration tests are run on AWS and the issue you describe occurs in Azure. I would consider this a primary issue as it is difficult for me to verify the behaviour you describe and fix it with a failing integration test at the start.\n\nAre you able to reproduce the issue on AWS Databricks environment so that we could include it in our integration tests and make sure the behvaiour will not change later on in future?" + } + ] + } + ] + } + ], + "team": "T01CWUYP5AR", + "thread_ts": "1699863517.394909", + "parent_user_id": "U05T8BJD4DU" + }, + { + "client_msg_id": "2f6a95e7-dd6d-45f4-8c79-119c1d004c1c", + "type": "message", + "text": "I didn't know Azure and AWS Databricks are different. Let me try it on AWS as well", + "user": "U05T8BJD4DU", + "ts": "1699916804.305259", + "blocks": [ + { + "type": "rich_text", + "block_id": "SAyd2", + "elements": [ + { + "type": "rich_text_section", + "elements": [ + { + "type": "text", + "text": "I didn't know Azure and AWS Databricks are different. Let me try it on AWS as well" + } + ] + } + ] + } + ], + "team": "T01CWUYP5AR", + "thread_ts": "1699863517.394909", + "parent_user_id": "U05T8BJD4DU" + } + ] + }, + { + "type": "message", + "text": "<@U02MK6YNAQ5> I went back to 1.4.1, output does show adls location. But environment facet is gone in 1.4.1. It shows up in 1.5.0 but namespace is back to dbfs....", + "files": [ + { + "id": "F0663EAEL0Y", + "created": 1699862379, + "timestamp": 1699862379, + "name": "data (11).json", + "title": "data (11).json", + "mimetype": "text/plain", + "filetype": "json", + "pretty_type": "JSON", + "user": "U05T8BJD4DU", + "user_team": "T01CWUYP5AR", + "editable": true, + "size": 52897, + "mode": "snippet", + "is_external": false, + "external_type": "", + "is_public": true, + "public_url_shared": false, + "display_as_bot": false, + "username": "", + "url_private": "https://files.slack.com/files-pri/T01CWUYP5AR-F0663EAEL0Y/data__11_.json", + "url_private_download": "https://files.slack.com/files-pri/T01CWUYP5AR-F0663EAEL0Y/download/data__11_.json", + "permalink": "https://openlineage.slack.com/files/U05T8BJD4DU/F0663EAEL0Y/data__11_.json", + "permalink_public": "https://slack-files.com/T01CWUYP5AR-F0663EAEL0Y-829953777d", + "edit_link": "https://openlineage.slack.com/files/U05T8BJD4DU/F0663EAEL0Y/data__11_.json/edit", + "preview": "{\r\n \"eventTime\":\"2023-11-13T07:49:59.575Z\",\r\n \"producer\":\"https://github.com/OpenLineage/OpenLineage/tree/1.4.1/integration/spark\",\r\n \"schemaURL\":\"https://openlineage.io/spec/2-0-2/OpenLineage.json#/$defs/RunEvent\",\r\n \"eventType\":\"COMPLETE\",\r", + "preview_highlight": "
\n
\n
{
\n
   "eventTime":"2023-11-13T07:49:59.575Z",
\n
   "producer":"https://github.com/OpenLineage/OpenLineage/tree/1.4.1/integration/spark",
\n
   "schemaURL":"https://openlineage.io/spec/2-0-2/OpenLineage.json#/$defs/RunEvent",
\n
   "eventType":"COMPLETE",
\n
\n
\n", + "lines": 1137, + "lines_more": 1132, + "preview_is_truncated": true, + "is_starred": false, + "has_rich_preview": false, + "file_access": "visible" + } + ], + "upload": false, + "user": "U05T8BJD4DU", + "display_as_bot": false, + "ts": "1699862442.876219", + "blocks": [ + { + "type": "rich_text", + "block_id": "RX5T+", + "elements": [ + { + "type": "rich_text_section", + "elements": [ + { + "type": "user", + "user_id": "U02MK6YNAQ5" + }, + { + "type": "text", + "text": " I went back to 1.4.1, output does show adls location. But environment facet is gone in 1.4.1. It shows up in 1.5.0 but namespace is back to dbfs...." + } + ] + } + ] + } + ], + "client_msg_id": "cdc89194-cce0-4480-8613-66db99259d74" + }, + { + "client_msg_id": "bc67ec9b-3910-4540-bcc0-eda4cf7452f1", + "type": "message", + "text": "Databricks needs to be re-written in a way that supports Databricks it seems like", + "user": "U05T8BJD4DU", + "ts": "1699691444.226989", + "blocks": [ + { + "type": "rich_text", + "block_id": "hkcRK", + "elements": [ + { + "type": "rich_text_section", + "elements": [ + { + "type": "text", + "text": "Databricks needs to be re-written in a way that supports Databricks it seems like" + } + ] + } + ] + } + ], + "team": "T01CWUYP5AR" + }, + { + "client_msg_id": "d06c50d8-7cdc-40c2-b20a-f3b375abbffe", + "type": "message", + "text": "<@U02MK6YNAQ5> this is why if create a table with adls location it won't show input and output:\n\n\n\nBecause the catalog object is not there.", + "user": "U05T8BJD4DU", + "ts": "1699691373.531469", + "blocks": [ + { + "type": "rich_text", + "block_id": "+2vNS", + "elements": [ + { + "type": "rich_text_section", + "elements": [ + { + "type": "user", + "user_id": "U02MK6YNAQ5" + }, + { + "type": "text", + "text": " this is why if create a table with adls location it won't show input and output:\n\n" + }, + { + "type": "link", + "url": "https://github.com/OpenLineage/OpenLineage/blob/main/integration/spark/spark35/src/main/java/io/openlineage/spark35/agent/lifecycle/plan/CreateReplaceOutputDatasetBuilder.java#L146-L148", + "text": "https://github.com/OpenLineage/OpenLineage/blob/main/integration/spark/spark35/src[…]k35/agent/lifecycle/plan/CreateReplaceOutputDatasetBuilder.java" + }, + { + "type": "text", + "text": "\n\nBecause the catalog object is not there." + } + ] + } + ] + } + ], + "team": "T01CWUYP5AR", + "edited": { + "user": "U05T8BJD4DU", + "ts": "1699691398.000000" + }, + "attachments": [ + { + "id": 1, + "footer_icon": "https://slack.github.com/static/img/favicon-neutral.png", + "color": "24292f", + "bot_id": "B01VA0FB340", + "app_unfurl_url": "https://github.com/OpenLineage/OpenLineage/blob/main/integration/spark/spark35/src/main/java/io/openlineage/spark35/agent/lifecycle/plan/CreateReplaceOutputDatasetBuilder.java#L146-L148", + "is_app_unfurl": true, + "app_id": "A01BP7R4KNY", + "fallback": "", + "text": "```\n if (!di.isPresent()) {\n return Collections.emptyList();\n }\n```", + "title": "", + "footer": "", + "mrkdwn_in": [ + "text" + ] + } + ] + }, + { + "client_msg_id": "bf43c6e5-d1ea-4f72-a555-937229c6e5f5", + "type": "message", + "text": "<@U02MK6YNAQ5> regarding to , OL is parsing out the table location in Hive metastore, it is the location of the table in the catalog and not the physical location of the data. It is both right and wrong because it is a table, just it is an external table.\n\n", + "user": "U05T8BJD4DU", + "ts": "1699647945.224489", + "blocks": [ + { + "type": "rich_text", + "block_id": "uqacZ", + "elements": [ + { + "type": "rich_text_section", + "elements": [ + { + "type": "user", + "user_id": "U02MK6YNAQ5" + }, + { + "type": "text", + "text": " regarding to " + }, + { + "type": "link", + "url": "https://github.com/OpenLineage/OpenLineage/issues/2124" + }, + { + "type": "text", + "text": ", OL is parsing out the table location in Hive metastore, it is the location of the table in the catalog and not the physical location of the data. It is both right and wrong because it is a table, just it is an external table.\n\n" + }, + { + "type": "link", + "url": "https://docs.databricks.com/en/sql/language-manual/sql-ref-external-tables.html" + } + ] + } + ] + } + ], + "team": "T01CWUYP5AR", + "edited": { + "user": "U05T8BJD4DU", + "ts": "1699647973.000000" + }, + "attachments": [ + { + "from_url": "https://docs.databricks.com/en/sql/language-manual/sql-ref-external-tables.html", + "image_url": "https://www.databricks.com/wp-content/uploads/2020/04/og-databricks.png", + "image_width": 1200, + "image_height": 630, + "image_bytes": 12097, + "service_icon": "https://docs.databricks.com/en/_static/favicon.ico", + "id": 1, + "original_url": "https://docs.databricks.com/en/sql/language-manual/sql-ref-external-tables.html", + "fallback": "External tables", + "text": "Learn about Unity Catalog external tables in Databricks SQL and Databricks Runtime.", + "title": "External tables", + "title_link": "https://docs.databricks.com/en/sql/language-manual/sql-ref-external-tables.html", + "service_name": "docs.databricks.com" + }, + { + "id": 2, + "footer_icon": "https://slack.github.com/static/img/favicon-neutral.png", + "ts": 1695498902, + "color": "36a64f", + "bot_id": "B01VA0FB340", + "app_unfurl_url": "https://github.com/OpenLineage/OpenLineage/issues/2124", + "is_app_unfurl": true, + "app_id": "A01BP7R4KNY", + "fallback": "#2124 Same Delta Table not catching the location on write", + "text": "*What is the target system?*\n\nSpark / Databricks\n\n*What kind of integration is this?*\n\n☐ Produces OpenLineage metadata\n☐ Consumes OpenLineage metadata\n☐ Something else\n\n*How should this integration be implemented?*\n\nI am using OL 1.2.2, Azure Databricks Runtime 11.3 LTS. When creating a table writing into a ADLS location, OL won't be able to catch the location of the output. But when I read the same object it will be able to read the location as INPUT.\n\nPlease note I have also tested Databricks Runtime 13.3 LTS, Spark 3.4.1 - it will give correct ADLS location in INPUT but the input will only show up once in a blue moon. Most of the time the inputs and outputs are blank.\n\n```\n \"inputs\": [],\n \"outputs\": []\n```\n\n```\nCREATE OR REPLACE TABLE transactions_adj\nUSING DELTA LOCATION ''\nAS\n SELECT\n household_id,\n basket_id,\n week_no,\n day,\n transaction_time,\n store_id,\n product_id,\n amount_list,\n campaign_coupon_discount,\n manuf_coupon_discount,\n manuf_coupon_match_discount,\n total_coupon_discount,\n instore_discount,\n amount_paid,\n units\n FROM (\n SELECT \n household_id,\n basket_id,\n week_no,\n day,\n transaction_time,\n store_id,\n product_id,\n COALESCE(sales_amount - discount_amount - coupon_discount - coupon_discount_match,0.0) as amount_list,\n CASE \n WHEN COALESCE(coupon_discount_match,0.0) = 0.0 THEN -1 * COALESCE(coupon_discount,0.0) \n ELSE 0.0 \n END as campaign_coupon_discount,\n CASE \n WHEN COALESCE(coupon_discount_match,0.0) != 0.0 THEN -1 * COALESCE(coupon_discount,0.0) \n ELSE 0.0 \n END as manuf_coupon_discount,\n -1 * COALESCE(coupon_discount_match,0.0) as manuf_coupon_match_discount,\n -1 * COALESCE(coupon_discount - coupon_discount_match,0.0) as total_coupon_discount,\n COALESCE(-1 * discount_amount,0.0) as instore_discount,\n COALESCE(sales_amount,0.0) as `amount_paid,`\n quantity as units\n FROM transactions\n );\n```\n\nHere's the COMPLETE event:\n\n```\n\n \"outputs\":[\n {\n \"namespace\":\"dbfs\",\n \"name\":\"/user/hive/warehouse/journey.db/transactions_adj\",\n \"facets\":{\n \"dataSource\":{\n \"_producer\":\"\",\n \"_schemaURL\":\"\",\n \"name\":\"dbfs\",\n \"uri\":\"dbfs\"\n },\n\n```\n\nBelow logical plan shows the path:\n\n```\n== Analyzed Logical Plan ==\nnum_affected_rows: bigint, num_inserted_rows: bigint\nReplaceTableAsSelect TableSpec(Map(),Some(DELTA),Map(),Some(),None,None,false,Set()), true\n:- ResolvedIdentifier com.databricks.sql.managedcatalog.UnityCatalogV2Proxy@6251a8df, default.transactions_adj\n+- Project [household_id#184, basket_id#185L, week_no#193, day#186, transaction_time#192, store_id#190, product_id#187, amount_list#147, campaign_coupon_discount#148, manuf_coupon_discount#149, manuf_coupon_match_discount#150, total_coupon_discount#151, instore_discount#152, amount_paid#153, units#154]\n +- SubqueryAlias __auto_generated_subquery_name\n +- Project [household_id#184, basket_id#185L, week_no#193, day#186, transaction_time#192, store_id#190, product_id#187, coalesce(cast((((sales_amount#189 - discount_amount#191) - coupon_discount#194) - coupon_discount_match#195) as double), cast(0.0 as double)) AS amount_list#147, CASE WHEN (coalesce(cast(coupon_discount_match#195 as double), cast(0.0 as double)) = cast(0.0 as double)) THEN (cast(-1 as double) * coalesce(cast(coupon_discount#194 as double), cast(0.0 as double))) ELSE cast(0.0 as double) END AS campaign_coupon_discount#148, CASE WHEN NOT (coalesce(cast(coupon_discount_match#195 as double), cast(0.0 as double)) = cast(0.0 as double)) THEN (cast(-1 as double) * coalesce(cast(coupon_discount#194 as double), cast(0.0 as double))) ELSE cast(0.0 as double) END AS manuf_coupon_discount#149, (cast(-1 as double) * coalesce(cast(coupon_discount_match#195 as double), cast(0.0 as double))) AS manuf_coupon_match_discount#150, (cast(-1 as double) * coalesce(cast((coupon_discount#194 - coupon_discount_match#195) as double), cast(0.0 as double))) AS total_coupon_discount#151, coalesce(cast((cast(-1 as float) * discount_amount#191) as double), cast(0.0 as double)) AS instore_discount#152, coalesce(cast(sales_amount#189 as double), cast(0.0 as double)) AS amount_paid#153, quantity#188 AS units#154]\n +- SubqueryAlias spark_catalog.default.transactions\n +- Relation spark_catalog.default.transactions[household_id#184,basket_id#185L,day#186,product_id#187,quantity#188,sales_amount#189,store_id#190,discount_amount#191,transaction_time#192,week_no#193,coupon_discount#194,coupon_discount_match#195] parquet\n```\n\n*Where should this integration be implemented?*\n\n☐ In the target system\n☐ In the OpenLineage repo\n☐ Somewhere else\n\n*Do you plan to make this contribution yourself?*\n\n☐ I am interested in doing this work", + "title": "#2124 Same Delta Table not catching the location on write", + "title_link": "https://github.com/OpenLineage/OpenLineage/issues/2124", + "footer": "", + "fields": [ + { + "value": "integration/spark, integration/databricks", + "title": "Labels", + "short": true + }, + { + "value": "5", + "title": "Comments", + "short": true + } + ], + "mrkdwn_in": [ + "text" + ] + } + ], + "thread_ts": "1699647945.224489", + "reply_count": 1, + "reply_users_count": 1, + "latest_reply": "1699648348.285049", + "reply_users": [ + "U05T8BJD4DU" + ], + "is_locked": false, + "subscribed": false, + "replies": [ + { + "client_msg_id": "34c0c0bc-49dd-48b6-9070-10348752259b", + "type": "message", + "text": "Here's for more reference: ", + "user": "U05T8BJD4DU", + "ts": "1699648348.285049", + "blocks": [ + { + "type": "rich_text", + "block_id": "XWYlR", + "elements": [ + { + "type": "rich_text_section", + "elements": [ + { + "type": "text", + "text": "Here's for more reference: " + }, + { + "type": "link", + "url": "https://dilorom.medium.com/finding-the-path-to-a-table-in-databricks-2c74c6009dbb" + } + ] + } + ] + } + ], + "team": "T01CWUYP5AR", + "attachments": [ + { + "from_url": "https://dilorom.medium.com/finding-the-path-to-a-table-in-databricks-2c74c6009dbb", + "ts": 1676841924, + "image_url": "https://miro.medium.com/v2/resize:fit:1200/0*mynRvuX_x2XOkqmx", + "image_width": 1200, + "image_height": 750, + "image_bytes": 196986, + "service_icon": "https://miro.medium.com/v2/resize:fill:152:152/1*sHhtYhaCe2Uc3IU0IgKwIQ.png", + "id": 1, + "original_url": "https://dilorom.medium.com/finding-the-path-to-a-table-in-databricks-2c74c6009dbb", + "fallback": "Medium: Finding the Path to a Managed Table in Databricks", + "text": "This article shows how to find a path for a managed Databricks table.", + "title": "Finding the Path to a Managed Table in Databricks", + "title_link": "https://dilorom.medium.com/finding-the-path-to-a-table-in-databricks-2c74c6009dbb", + "service_name": "Medium", + "fields": [ + { + "value": "2 min read", + "title": "Reading time", + "short": true + } + ] + } + ], + "thread_ts": "1699647945.224489", + "parent_user_id": "U05T8BJD4DU" + } + ] + }, + { + "client_msg_id": "7210d352-df88-4ba2-bebd-a47cbd15b2d4", + "type": "message", + "text": "\nFriendly reminder: this month’s TSC meeting, open to all, is tomorrow at 10 am PT: ", + "user": "U02LXF3HUN7", + "ts": "1699465494.687309", + "blocks": [ + { + "type": "rich_text", + "block_id": "OMGKh", + "elements": [ + { + "type": "rich_text_section", + "elements": [ + { + "type": "broadcast", + "range": "channel" + }, + { + "type": "text", + "text": "\nFriendly reminder: this month’s TSC meeting, open to all, is tomorrow at 10 am PT: " + }, + { + "type": "link", + "url": "https://openlineage.slack.com/archives/C01CK9T7HKR/p1699027207361229" + } + ] + } + ] + } + ], + "team": "T01CWUYP5AR", + "attachments": [ + { + "from_url": "https://openlineage.slack.com/archives/C01CK9T7HKR/p1699027207361229", + "ts": "1699027207.361229", + "author_id": "U02LXF3HUN7", + "channel_id": "C01CK9T7HKR", + "channel_team": "T01CWUYP5AR", + "is_msg_unfurl": true, + "message_blocks": [ + { + "team": "T01CWUYP5AR", + "channel": "C01CK9T7HKR", + "ts": "1699027207.361229", + "message": { + "blocks": [ + { + "type": "rich_text", + "block_id": "VnOMq", + "elements": [ + { + "type": "rich_text_section", + "elements": [ + { + "type": "broadcast", + "range": "channel" + }, + { + "type": "text", + "text": "\nThis month’s TSC meeting (open to all) is next Thursday the 9th at 10am PT. On the agenda:\n" + } + ] + }, + { + "type": "rich_text_list", + "elements": [ + { + "type": "rich_text_section", + "elements": [ + { + "type": "text", + "text": "announcements" + } + ] + }, + { + "type": "rich_text_section", + "elements": [ + { + "type": "text", + "text": "recent releases" + } + ] + }, + { + "type": "rich_text_section", + "elements": [ + { + "type": "text", + "text": "recent additions to the Flink integration by " + }, + { + "type": "user", + "user_id": "U05QA2D1XNV" + }, + { + "type": "text", + "text": " " + } + ] + }, + { + "type": "rich_text_section", + "elements": [ + { + "type": "text", + "text": "recent additions to the Spark integration by " + }, + { + "type": "user", + "user_id": "U02MK6YNAQ5" + }, + { + "type": "text", + "text": " " + } + ] + }, + { + "type": "rich_text_section", + "elements": [ + { + "type": "text", + "text": "updates on proposals by " + }, + { + "type": "user", + "user_id": "U01DCLP0GU9" + }, + { + "type": "text", + "text": " " + } + ] + }, + { + "type": "rich_text_section", + "elements": [ + { + "type": "text", + "text": "discussion topics" + } + ] + }, + { + "type": "rich_text_section", + "elements": [ + { + "type": "text", + "text": "open discussion" + } + ] + } + ], + "style": "bullet", + "indent": 0, + "border": 0 + }, + { + "type": "rich_text_section", + "elements": [ + { + "type": "text", + "text": "More info and the meeting link can be found on the " + }, + { + "type": "link", + "url": "https://openlineage.io/meetings/", + "text": "website" + }, + { + "type": "text", + "text": ". All are welcome! Do you have a discussion topic, use case or integration you’d like to demo? DM me to be added to the agenda." + } + ] + } + ] + } + ] + } + } + ], + "id": 1, + "original_url": "https://openlineage.slack.com/archives/C01CK9T7HKR/p1699027207361229", + "fallback": "[November 3rd, 2023 9:00 AM] michael282: \nThis month’s TSC meeting (open to all) is next Thursday the 9th at 10am PT. On the agenda:\n• announcements\n• recent releases\n• recent additions to the Flink integration by <@U05QA2D1XNV> \n• recent additions to the Spark integration by <@U02MK6YNAQ5> \n• updates on proposals by <@U01DCLP0GU9> \n• discussion topics\n• open discussion\nMore info and the meeting link can be found on the . All are welcome! Do you have a discussion topic, use case or integration you’d like to demo? DM me to be added to the agenda.", + "text": "\nThis month’s TSC meeting (open to all) is next Thursday the 9th at 10am PT. On the agenda:\n• announcements\n• recent releases\n• recent additions to the Flink integration by <@U05QA2D1XNV> \n• recent additions to the Spark integration by <@U02MK6YNAQ5> \n• updates on proposals by <@U01DCLP0GU9> \n• discussion topics\n• open discussion\nMore info and the meeting link can be found on the . All are welcome! Do you have a discussion topic, use case or integration you’d like to demo? DM me to be added to the agenda.", + "author_name": "Michael Robinson", + "author_link": "https://openlineage.slack.com/team/U02LXF3HUN7", + "author_icon": "https://avatars.slack-edge.com/2022-01-25/3019716733729_66fea720e9504dc08144_48.jpg", + "author_subname": "Michael Robinson", + "mrkdwn_in": [ + "text" + ], + "footer": "Slack Conversation" + } + ], + "reactions": [ + { + "name": "+1", + "users": [ + "U02S6F54MAB" + ], + "count": 1 + } + ] + }, + { + "client_msg_id": "425424cf-69e6-4b05-9699-130951f09ab6", + "type": "message", + "text": "Has anyone here tried OpenLineage with Spark on Amazon EMR?", + "user": "U05TU0U224A", + "ts": "1699465132.534889", + "blocks": [ + { + "type": "rich_text", + "block_id": "AHU5F", + "elements": [ + { + "type": "rich_text_section", + "elements": [ + { + "type": "text", + "text": "Has anyone here tried OpenLineage with Spark on Amazon EMR?" + } + ] + } + ] + } + ], + "team": "T01CWUYP5AR", + "thread_ts": "1699465132.534889", + "reply_count": 3, + "reply_users_count": 3, + "latest_reply": "1700497766.698579", + "reply_users": [ + "U05T8BJD4DU", + "U053LCT71BQ", + "U05TU0U224A" + ], + "is_locked": false, + "subscribed": false, + "replies": [ + { + "client_msg_id": "dcefe260-742d-4735-ad72-7e19d7303e07", + "type": "message", + "text": "No but it should work the same I tried on AWS and Google Colab and Azure", + "user": "U05T8BJD4DU", + "ts": "1699466476.239829", + "blocks": [ + { + "type": "rich_text", + "block_id": "DhY38", + "elements": [ + { + "type": "rich_text_section", + "elements": [ + { + "type": "text", + "text": "No but it should work the same I tried on AWS and Google Colab and Azure" + } + ] + } + ] + } + ], + "team": "T01CWUYP5AR", + "thread_ts": "1699465132.534889", + "parent_user_id": "U05TU0U224A", + "reactions": [ + { + "name": "+1", + "users": [ + "U02S6F54MAB" + ], + "count": 1 + } + ] + }, + { + "client_msg_id": "2bc40618-6fa1-4d3f-b5ce-37a92d166c2b", + "type": "message", + "text": "Yes. <@U05HBLE7YPL> could provide some details if needed.", + "user": "U053LCT71BQ", + "ts": "1699517454.058999", + "blocks": [ + { + "type": "rich_text", + "block_id": "/ny7q", + "elements": [ + { + "type": "rich_text_section", + "elements": [ + { + "type": "text", + "text": "Yes. " + }, + { + "type": "user", + "user_id": "U05HBLE7YPL" + }, + { + "type": "text", + "text": " could provide some details if needed." + } + ] + } + ] + } + ], + "team": "T01CWUYP5AR", + "thread_ts": "1699465132.534889", + "parent_user_id": "U05TU0U224A", + "reactions": [ + { + "name": "+1", + "users": [ + "U05HBLE7YPL" + ], + "count": 1 + }, + { + "name": "fire", + "users": [ + "U01RA9B5GG2" + ], + "count": 1 + } + ] + }, + { + "client_msg_id": "73fc95f8-a897-4a16-ae48-f494d98970eb", + "type": "message", + "text": "Thanks <@U053LCT71BQ>\nHI <@U05HBLE7YPL> i was able to set up a spark cluster on AWS EMR but im struggling to configure the OL Listener. Ive tried with steps and bootstrap actions for the jar and it didn't work out. How did you manage to include the jar? Besides, what about the spark configuration? Could you send me a sample of these configs?", + "user": "U05TU0U224A", + "ts": "1700497766.698579", + "blocks": [ + { + "type": "rich_text", + "block_id": "NuX5f", + "elements": [ + { + "type": "rich_text_section", + "elements": [ + { + "type": "text", + "text": "Thanks " + }, + { + "type": "user", + "user_id": "U053LCT71BQ" + }, + { + "type": "text", + "text": "\nHI " + }, + { + "type": "user", + "user_id": "U05HBLE7YPL" + }, + { + "type": "text", + "text": " i was able to set up a spark cluster on AWS EMR but im struggling to configure the OL Listener. Ive tried with steps and bootstrap actions for the jar and it didn't work out. How did you manage to include the jar? Besides, what about the spark configuration? Could you send me a sample of these configs?" + } + ] + } + ] + } + ], + "team": "T01CWUYP5AR", + "thread_ts": "1699465132.534889", + "parent_user_id": "U05TU0U224A" + } + ] + }, + { + "client_msg_id": "1cf56ca5-8527-4399-9b26-877012f4d625", + "type": "message", + "text": "if I have a dataset on adls gen2 which synapse connects to as an external delta table, is that the use case of a symlink dataset? the delta table is connected to by PBI and by Synapse, but the underlying data is exactly the same", + "user": "U05NMJ0NBUK", + "ts": "1699372165.804069", + "blocks": [ + { + "type": "rich_text", + "block_id": "rqeWx", + "elements": [ + { + "type": "rich_text_section", + "elements": [ + { + "type": "text", + "text": "if I have a dataset on adls gen2 which synapse connects to as an external delta table, is that the use case of a symlink dataset? the delta table is connected to by PBI and by Synapse, but the underlying data is exactly the same" + } + ] + } + ] + } + ], + "team": "T01CWUYP5AR", + "thread_ts": "1699372165.804069", + "reply_count": 1, + "reply_users_count": 1, + "latest_reply": "1699458544.315609", + "reply_users": [ + "U01RA9B5GG2" + ], + "is_locked": false, + "subscribed": false, + "replies": [ + { + "client_msg_id": "3d4517cc-ae51-40f5-a2de-5721ba9585be", + "type": "message", + "text": "Sounds like it, yes - if the logical dataset names are different but physical one is the same", + "user": "U01RA9B5GG2", + "ts": "1699458544.315609", + "blocks": [ + { + "type": "rich_text", + "block_id": "bEjsY", + "elements": [ + { + "type": "rich_text_section", + "elements": [ + { + "type": "text", + "text": "Sounds like it, yes - if the logical dataset names are different but physical one is the same" + } + ] + } + ] + } + ], + "team": "T01CWUYP5AR", + "thread_ts": "1699372165.804069", + "parent_user_id": "U05NMJ0NBUK" + } + ] + }, + { + "client_msg_id": "b227b661-6908-4ebb-9e0b-34e904a9dcdc", + "type": "message", + "text": "Hi all, we (I work with <@U05VDHJJ9T7> and <@U05HBLE7YPL>) have a quick question regarding the spark integration:\nif a spark app contains several jobs, they will be named \"my_spark_app_name.job1\" and \"my_spark_app_name.job2\"\neg:\nspark_job.collect_limit\nspark_job.map_partitions_parallel_collection\n\nIf I understood correctly, the spark integration maps one Spark job to a single OpenLineage Job, and the application itself should be assigned a Run id at startup and each job that executes will report the application's Run id as its parent job run (taken from: ).\n\nIn our case, the app Run Id is never created, and the jobs runs don't contain any parent facets. We tested it with a recent integration version in 1.4.1 and also an older one (0.26.0).\nDid we miss something in the OL spark integration config?", + "user": "U05J9LZ355L", + "ts": "1699355029.839029", + "blocks": [ + { + "type": "rich_text", + "block_id": "qYUfo", + "elements": [ + { + "type": "rich_text_section", + "elements": [ + { + "type": "text", + "text": "Hi all, we (I work with " + }, + { + "type": "user", + "user_id": "U05VDHJJ9T7" + }, + { + "type": "text", + "text": " and " + }, + { + "type": "user", + "user_id": "U05HBLE7YPL" + }, + { + "type": "text", + "text": ") have a quick question regarding the spark integration:\nif a spark app contains several jobs, they will be named \"my_spark_app_name.job1\" and \"my_spark_app_name.job2\"\neg:\nspark_job.collect_limit\nspark_job.map_partitions_parallel_collection\n\nIf I understood correctly, the spark integration maps one Spark job to a single OpenLineage Job, and the application itself should be assigned a Run id at startup and each job that executes will report the application's Run id as its parent job run (taken from: " + }, + { + "type": "link", + "url": "https://openlineage.io/docs/integrations/spark/" + }, + { + "type": "text", + "text": ").\n\nIn our case, the app Run Id is never created, and the jobs runs don't contain any parent facets. We tested it with a recent integration version in 1.4.1 and also an older one (0.26.0).\nDid we miss something in the OL spark integration config?" + } + ] + } + ] + } + ], + "team": "T01CWUYP5AR", + "thread_ts": "1699355029.839029", + "reply_count": 4, + "reply_users_count": 2, + "latest_reply": "1699364218.932689", + "reply_users": [ + "U02MK6YNAQ5", + "U05J9LZ355L" + ], + "is_locked": false, + "subscribed": false, + "replies": [ + { + "client_msg_id": "4bb63de7-60e0-4443-bc53-395529786d26", + "type": "message", + "text": "hey, a name of the output dataset should be put at the end of the job name. This was introduced to help with jobs that call multiple spark actions", + "user": "U02MK6YNAQ5", + "ts": "1699355271.048109", + "blocks": [ + { + "type": "rich_text", + "block_id": "11PhC", + "elements": [ + { + "type": "rich_text_section", + "elements": [ + { + "type": "text", + "text": "hey, a name of the output dataset should be put at the end of the job name. This was introduced to help with jobs that call multiple spark actions" + } + ] + } + ] + } + ], + "team": "T01CWUYP5AR", + "thread_ts": "1699355029.839029", + "parent_user_id": "U05J9LZ355L" + }, + { + "client_msg_id": "93865d43-03ba-4aed-bc8e-d134ea79539d", + "type": "message", + "text": "Hi Paweł,\nThanks for your answer, yes indeed with the newer version of OL, we automatically have the name of the output dataset at the end of the job name, but no App run id, nor any parent run facet.", + "user": "U05J9LZ355L", + "ts": "1699358752.643889", + "blocks": [ + { + "type": "rich_text", + "block_id": "Saf+l", + "elements": [ + { + "type": "rich_text_section", + "elements": [ + { + "type": "text", + "text": "Hi Paweł,\nThanks for your answer, yes indeed with the newer version of OL, we automatically have the name of the output dataset at the end of the job name, but no App run id, nor any parent run facet." + } + ] + } + ] + } + ], + "team": "T01CWUYP5AR", + "thread_ts": "1699355029.839029", + "parent_user_id": "U05J9LZ355L" + }, + { + "client_msg_id": "f77ea12e-ea22-4c07-a1a9-8e7a2b45c9a3", + "type": "message", + "text": "yes, you're right. I mean you can set in config `spark.openlineage.parentJobName` which will be shared through whole app run, but this needs to be set manually", + "user": "U02MK6YNAQ5", + "ts": "1699363004.077549", + "blocks": [ + { + "type": "rich_text", + "block_id": "L5WWY", + "elements": [ + { + "type": "rich_text_section", + "elements": [ + { + "type": "text", + "text": "yes, you're right. I mean you can set in config " + }, + { + "type": "text", + "text": "spark.openlineage.parentJobName", + "style": { + "code": true + } + }, + { + "type": "text", + "text": " which will be shared through whole app run, but this needs to be set manually" + } + ] + } + ] + } + ], + "team": "T01CWUYP5AR", + "thread_ts": "1699355029.839029", + "parent_user_id": "U05J9LZ355L" + }, + { + "client_msg_id": "24dc4817-ff81-4505-b814-0c6090f0301d", + "type": "message", + "text": "I see, thanks a lot for your reply we'll try that", + "user": "U05J9LZ355L", + "ts": "1699364218.932689", + "blocks": [ + { + "type": "rich_text", + "block_id": "b8b1A", + "elements": [ + { + "type": "rich_text_section", + "elements": [ + { + "type": "text", + "text": "I see, thanks a lot for your reply we'll try that" + } + ] + } + ] + } + ], + "team": "T01CWUYP5AR", + "thread_ts": "1699355029.839029", + "parent_user_id": "U05J9LZ355L" + } + ] + }, + { + "client_msg_id": "4f10979d-6d7d-4a68-8a53-00a158a9c222", + "type": "message", + "text": "Hey team! :wave:\n\nWe're trying to use openlineage-flink, and would like provide the `openlineage.transport.type=http` and configure other transport configs, but we're not able to find sufficient docs (tried ) on where/how these configs can be provided.\n\nFor example, in spark, the changes mostly were delegated to the spark-submit command like\n```spark-submit --conf \"spark.extraListeners=io.openlineage.spark.agent.OpenLineageSparkListener\" \\\n --packages \"io.openlineage:openlineage-spark:<spark-openlineage-version>\" \\\n --conf \"spark.openlineage.transport.url=http://{openlineage.client.host}/api/v1/namespaces/spark_integration/\" \\\n --class com.mycompany.MySparkApp my_application.jar```\nAnd the `OpenLineageSparkListener` has a method to retrieve the provided spark confs as an object in the ArgumentParser. Similarly, looking for some pointers on how the openlineage.transport configs can be provided to `OpenLineageFlinkJobListener` & how the flink listener parses/uses these configs\n\nTIA! :smile:", + "user": "U05JBHLPY8K", + "ts": "1699266123.453379", + "blocks": [ + { + "type": "rich_text", + "block_id": "77vAn", + "elements": [ + { + "type": "rich_text_section", + "elements": [ + { + "type": "text", + "text": "Hey team! " + }, + { + "type": "emoji", + "name": "wave", + "unicode": "1f44b" + }, + { + "type": "text", + "text": "\n\nWe're trying to use openlineage-flink, and would like provide the " + }, + { + "type": "text", + "text": "openlineage.transport.type=http", + "style": { + "code": true + } + }, + { + "type": "text", + "text": " and configure other transport configs, but we're not able to find sufficient docs (tried " + }, + { + "type": "link", + "url": "https://openlineage.io/docs/integrations/flink", + "text": "this doc" + }, + { + "type": "text", + "text": ") on where/how these configs can be provided.\n\nFor example, in spark, the changes mostly were delegated to the spark-submit command like\n" + } + ] + }, + { + "type": "rich_text_preformatted", + "elements": [ + { + "type": "text", + "text": "spark-submit --conf \"spark.extraListeners=io.openlineage.spark.agent.OpenLineageSparkListener\" \\\n --packages \"io.openlineage:openlineage-spark:\" \\\n --conf \"spark.openlineage.transport.url=http://{openlineage.client.host}/api/v1/namespaces/spark_integration/\" \\\n --class com.mycompany.MySparkApp my_application.jar" + } + ], + "border": 0 + }, + { + "type": "rich_text_section", + "elements": [ + { + "type": "text", + "text": "\nAnd the " + }, + { + "type": "text", + "text": "OpenLineageSparkListener", + "style": { + "code": true + } + }, + { + "type": "text", + "text": " has a method to retrieve the provided spark confs as an object in the ArgumentParser. Similarly, looking for some pointers on how the openlineage.transport configs can be provided to " + }, + { + "type": "text", + "text": "OpenLineageFlinkJobListener", + "style": { + "code": true + } + }, + { + "type": "text", + "text": " & how the flink listener parses/uses these configs\n\nTIA! " + }, + { + "type": "emoji", + "name": "smile", + "unicode": "1f604" + } + ] + } + ] + } + ], + "team": "T01CWUYP5AR", + "attachments": [ + { + "from_url": "https://openlineage.io/docs/integrations/flink", + "service_icon": "https://openlineage.io/img/favicon.ico", + "id": 1, + "original_url": "https://openlineage.io/docs/integrations/flink", + "fallback": "Apache Flink | OpenLineage", + "text": "This integration is considered experimental: only specific workflows and use cases are supported.", + "title": "Apache Flink | OpenLineage", + "title_link": "https://openlineage.io/docs/integrations/flink", + "service_name": "openlineage.io" + } + ], + "thread_ts": "1699266123.453379", + "reply_count": 5, + "reply_users_count": 2, + "latest_reply": "1699553036.057049", + "reply_users": [ + "U01RA9B5GG2", + "U05JBHLPY8K" + ], + "is_locked": false, + "subscribed": false, + "replies": [ + { + "client_msg_id": "b718b130-01f6-4c72-97a8-2145978818f4", + "type": "message", + "text": "similarly to spark config, you can use flink config", + "user": "U01RA9B5GG2", + "ts": "1699354569.864879", + "blocks": [ + { + "type": "rich_text", + "block_id": "3gH65", + "elements": [ + { + "type": "rich_text_section", + "elements": [ + { + "type": "text", + "text": "similarly to spark config, you can use flink config" + } + ] + } + ] + } + ], + "team": "T01CWUYP5AR", + "thread_ts": "1699266123.453379", + "parent_user_id": "U05JBHLPY8K" + }, + { + "client_msg_id": "767B5029-9253-418F-91D8-1B355C1CB159", + "type": "message", + "text": "<@U01RA9B5GG2> - Got it. Our use-case is that we're trying to build a wrapper on top of openlineage-flink for productionising for our flink jobs.\n\nWe're trying to have a wrapper class that extends OpenLineageFlinkJobListener class, and overwrites the HTTP transport endpoint/url to a constant value (say, and /api/v1/flink). But we see that the OpenLineageFlinkJobListener constructor is defined as a private constructor - just wanted to check with the team whether it was just a default scope, or intended to be private. If it was just a default scope, can we contribute a PR to make it public, to make it friendly for teams trying to adopt & extend openlineage?\n\nAnd also, we wanted to understand better on where we're reading the HTTP transport endpoint/url configs in OpenLineageFlinkJobListener and what'd be the best place to override it to the constant endpoint/url for our use-case", + "user": "U05JBHLPY8K", + "ts": "1699414613.261979", + "blocks": [ + { + "type": "rich_text", + "block_id": "U0Vuf", + "elements": [ + { + "type": "rich_text_section", + "elements": [ + { + "type": "user", + "user_id": "U01RA9B5GG2" + }, + { + "type": "text", + "text": " - Got it. Our use-case is that we're trying to build a wrapper on top of openlineage-flink for productionising for our flink jobs.\n\nWe're trying to have a wrapper class that extends OpenLineageFlinkJobListener class, and overwrites the HTTP transport endpoint/url to a constant value (say, " + }, + { + "type": "link", + "url": "http://example.com", + "text": "example.com" + }, + { + "type": "text", + "text": " and /api/v1/flink). But we see that the OpenLineageFlinkJobListener constructor is defined as a private constructor - just wanted to check with the team whether it was just a default scope, or intended to be private. If it was just a default scope, can we contribute a PR to make it public, to make it friendly for teams trying to adopt & extend openlineage?\n\nAnd also, we wanted to understand better on where we're reading the HTTP transport endpoint/url configs in OpenLineageFlinkJobListener and what'd be the best place to override it to the constant endpoint/url for our use-case" + } + ] + } + ] + } + ], + "team": "T01CWUYP5AR", + "edited": { + "user": "U05JBHLPY8K", + "ts": "1699415789.000000" + }, + "thread_ts": "1699266123.453379", + "parent_user_id": "U05JBHLPY8K" + }, + { + "client_msg_id": "5d48005e-a7c3-43ad-8f94-26e47ceddba5", + "type": "message", + "text": "We parse flink conf to get that information: \n\n> But we see that the OpenLineageFlinkJobListener constructor is defined as a private constructor - just wanted to check with the team whether it was just a default scope, or intended to be private.\nThe way to construct is is a public builder in the same class\n\nI think easier way than wrapper class would be use existing flink configuration, or to set up `OPENLINEAGE_URL` env variable, or have `openlineage.yml` config file - not sure why this is the way you've chosen?", + "user": "U01RA9B5GG2", + "ts": "1699440943.234349", + "blocks": [ + { + "type": "rich_text", + "block_id": "mBhxR", + "elements": [ + { + "type": "rich_text_section", + "elements": [ + { + "type": "text", + "text": "We parse flink conf to get that information: " + }, + { + "type": "link", + "url": "https://github.com/OpenLineage/OpenLineage/blob/26494b596e9669d2ada164066a73c44e04e884ba/integration/flink/src/main/java/io/openlineage/flink/client/EventEmitter.java#L35", + "text": "https://github.com/OpenLineage/OpenLineage/blob/26494b596e9669d2ada164066a73c44e04[…]ink/src/main/java/io/openlineage/flink/client/EventEmitter.java" + }, + { + "type": "text", + "text": "\n\n" + } + ] + }, + { + "type": "rich_text_quote", + "elements": [ + { + "type": "text", + "text": "But we see that the OpenLineageFlinkJobListener constructor is defined as a private constructor - just wanted to check with the team whether it was just a default scope, or intended to be private." + } + ] + }, + { + "type": "rich_text_section", + "elements": [ + { + "type": "text", + "text": "\nThe way to construct is is a public builder in the same class\n\nI think easier way than wrapper class would be use existing flink configuration, or to set up " + }, + { + "type": "text", + "text": "OPENLINEAGE_URL", + "style": { + "code": true + } + }, + { + "type": "text", + "text": " env variable, or have " + }, + { + "type": "text", + "text": "openlineage.yml", + "style": { + "code": true + } + }, + { + "type": "text", + "text": " config file - not sure why this is the way you've chosen?" + } + ] + } + ] + } + ], + "team": "T01CWUYP5AR", + "attachments": [ + { + "id": 1, + "footer_icon": "https://slack.github.com/static/img/favicon-neutral.png", + "color": "24292f", + "bot_id": "B01VA0FB340", + "app_unfurl_url": "https://github.com/OpenLineage/OpenLineage/blob/26494b596e9669d2ada164066a73c44e04e884ba/integration/flink/src/main/java/io/openlineage/flink/client/EventEmitter.java#L35", + "is_app_unfurl": true, + "app_id": "A01BP7R4KNY", + "fallback": "", + "text": "```\n public EventEmitter(Configuration configuration) {\n```", + "title": "", + "footer": "", + "mrkdwn_in": [ + "text" + ] + } + ], + "thread_ts": "1699266123.453379", + "parent_user_id": "U05JBHLPY8K" + }, + { + "client_msg_id": "665487f6-7eb8-4b61-b441-b1f00cbf90b9", + "type": "message", + "text": "> I think easier way than wrapper class would be use existing flink configuration, or to set up `OPENLINEAGE_URL` env variable, or have `openlineage.yml` config file - not sure why this is the way you've chosen?\n<@U01RA9B5GG2> - The reasoning behind going with a wrapper class is that we can abstract out the nitty-gritty like how/where we're publishing openlineage events etc - especially for companies that have a lot of teams that may be adopting openlineage.\n\nFor example, if we wanna move away from http transport to kafka transport - we'd be changing only this wrapper class and ask folks to update their wrapper class dependency version. If we went without the wrapper class, then the exact config changes would need to be synced and done by many different teams, who may not have enough context.\n\nSimilarly, if we wanna enable some other default best-practise configs, or inject any company-specific configs etc, the wrapper would be useful in abstracting out the details and be the 1 place that handles all openlineage related integrations for any future changes.\n\nThat's why we wanna extend openlineage's listener class & leverage most of the OSS code as-is; and at the same time, have the ability to extend & inject customisations. I think that's where some things like having getters for the class object attributes, or having public constructors would be really helpful :smile:", + "user": "U05JBHLPY8K", + "ts": "1699551662.975039", + "blocks": [ + { + "type": "rich_text", + "block_id": "eIuSR", + "elements": [ + { + "type": "rich_text_quote", + "elements": [ + { + "type": "text", + "text": "I think easier way than wrapper class would be use existing flink configuration, or to set up " + }, + { + "type": "text", + "text": "OPENLINEAGE_URL", + "style": { + "code": true + } + }, + { + "type": "text", + "text": " env variable, or have " + }, + { + "type": "text", + "text": "openlineage.yml", + "style": { + "code": true + } + }, + { + "type": "text", + "text": " config file - not sure why this is the way you've chosen?" + } + ] + }, + { + "type": "rich_text_section", + "elements": [ + { + "type": "text", + "text": "\n" + }, + { + "type": "user", + "user_id": "U01RA9B5GG2" + }, + { + "type": "text", + "text": " - The reasoning behind going with a wrapper class is that we can abstract out the nitty-gritty like how/where we're publishing openlineage events etc - especially for companies that have a lot of teams that may be adopting openlineage.\n\nFor example, if we wanna move away from http transport to kafka transport - we'd be changing only this wrapper class and ask folks to update their wrapper class dependency version. If we went without the wrapper class, then the exact config changes would need to be synced and done by many different teams, who may not have enough context.\n\nSimilarly, if we wanna enable some other default best-practise configs, or inject any company-specific configs etc, the wrapper would be useful in abstracting out the details and be the 1 place that handles all openlineage related integrations for any future changes.\n\nThat's why we wanna extend openlineage's listener class & leverage most of the OSS code as-is; and at the same time, have the ability to extend & inject customisations. I think that's where some things like having getters for the class object attributes, or having public constructors would be really helpful " + }, + { + "type": "emoji", + "name": "smile", + "unicode": "1f604" + } + ] + } + ] + } + ], + "team": "T01CWUYP5AR", + "edited": { + "user": "U05JBHLPY8K", + "ts": "1699555439.000000" + }, + "thread_ts": "1699266123.453379", + "parent_user_id": "U05JBHLPY8K" + }, + { + "client_msg_id": "e0063ce9-8e38-4649-8a66-60d769bee3fa", + "type": "message", + "text": "<@U05JBHLPY8K> that makes sense. Feel free to provide PR adding getters and stuff.", + "user": "U01RA9B5GG2", + "ts": "1699553036.057049", + "blocks": [ + { + "type": "rich_text", + "block_id": "xscjy", + "elements": [ + { + "type": "rich_text_section", + "elements": [ + { + "type": "user", + "user_id": "U05JBHLPY8K" + }, + { + "type": "text", + "text": " that makes sense. Feel free to provide PR adding getters and stuff." + } + ] + } + ] + } + ], + "team": "T01CWUYP5AR", + "thread_ts": "1699266123.453379", + "parent_user_id": "U05JBHLPY8K", + "reactions": [ + { + "name": "tada", + "users": [ + "U05JBHLPY8K" + ], + "count": 1 + } + ] + } + ] + }, + { + "client_msg_id": "9ddf3aa3-95fb-4922-94da-538fcfe0aeae", + "type": "message", + "text": ":wave: I raised a PR off the back of some Marquez conversations a while back to try and clarify how names of Snowflake objects should be expressed in OL events. I used as a guide, but also I appreciate there are other OL producers that involve Snowflake too (Airflow? dbt?). Any feedback on this would be appreciated!", + "user": "U0635GK8Y14", + "ts": "1699261422.618719", + "blocks": [ + { + "type": "rich_text", + "block_id": "PCfWI", + "elements": [ + { + "type": "rich_text_section", + "elements": [ + { + "type": "emoji", + "name": "wave", + "unicode": "1f44b" + }, + { + "type": "text", + "text": " I raised a PR " + }, + { + "type": "link", + "url": "https://github.com/OpenLineage/OpenLineage/pull/2223" + }, + { + "type": "text", + "text": " off the back of some Marquez conversations a while back to try and clarify how names of Snowflake objects should be expressed in OL events. I used " + }, + { + "type": "link", + "url": "https://github.com/Snowflake-Labs/OpenLineage-AccessHistory-Setup", + "text": "Snowflake’s OL view" + }, + { + "type": "text", + "text": " as a guide, but also I appreciate there are other OL producers that involve Snowflake too (Airflow? dbt?). Any feedback on this would be appreciated!" + } + ] + } + ] + } + ], + "team": "T01CWUYP5AR", + "attachments": [ + { + "id": 1, + "footer_icon": "https://slack.github.com/static/img/favicon-neutral.png", + "ts": 1698847613, + "color": "36a64f", + "bot_id": "B01VA0FB340", + "app_unfurl_url": "https://github.com/OpenLineage/OpenLineage/pull/2223", + "is_app_unfurl": true, + "app_id": "A01BP7R4KNY", + "fallback": "#2223 spec: add clarity to snowflake naming docs", + "title": "#2223 spec: add clarity to snowflake naming docs", + "title_link": "https://github.com/OpenLineage/OpenLineage/pull/2223", + "footer": "", + "mrkdwn_in": [ + "text" + ] + }, + { + "id": 2, + "color": "24292f", + "bot_id": "B01VA0FB340", + "app_unfurl_url": "https://github.com/Snowflake-Labs/OpenLineage-AccessHistory-Setup", + "is_app_unfurl": true, + "app_id": "A01BP7R4KNY", + "fallback": "Snowflake-Labs/OpenLineage-AccessHistory-Setup", + "title": "Snowflake-Labs/OpenLineage-AccessHistory-Setup", + "fields": [ + { + "value": "11", + "title": "Stars", + "short": true + }, + { + "value": "3 months ago", + "title": "Last updated", + "short": true + } + ] + } + ], + "thread_ts": "1699261422.618719", + "reply_count": 1, + "reply_users_count": 1, + "latest_reply": "1699458155.637699", + "reply_users": [ + "U0635GK8Y14" + ], + "is_locked": false, + "subscribed": false, + "replies": [ + { + "client_msg_id": "d6c8048f-0074-4940-9c8d-428c7f108399", + "type": "message", + "text": "Thanks for merging this <@U01RA9B5GG2>!", + "user": "U0635GK8Y14", + "ts": "1699458155.637699", + "blocks": [ + { + "type": "rich_text", + "block_id": "QoDq+", + "elements": [ + { + "type": "rich_text_section", + "elements": [ + { + "type": "text", + "text": "Thanks for merging this " + }, + { + "type": "user", + "user_id": "U01RA9B5GG2" + }, + { + "type": "text", + "text": "!" + } + ] + } + ] + } + ], + "team": "T01CWUYP5AR", + "thread_ts": "1699261422.618719", + "parent_user_id": "U0635GK8Y14", + "reactions": [ + { + "name": "+1", + "users": [ + "U01RA9B5GG2" + ], + "count": 1 + } + ] + } + ] + }, + { + "client_msg_id": "57c0c5ad-870f-4ea8-9d65-e86039204baa", + "type": "message", + "text": "Hi Team , we are trying to customize the events by writing custom lineage listener extending OpenLineageSparkListener, but would need some direction how to capture the events", + "user": "U062Q95A1FG", + "ts": "1699096090.087359", + "blocks": [ + { + "type": "rich_text", + "block_id": "rx6pv", + "elements": [ + { + "type": "rich_text_section", + "elements": [ + { + "type": "text", + "text": "Hi Team , we are trying to customize the events by writing custom lineage listener extending OpenLineageSparkListener, but would need some direction how to capture the events" + } + ] + } + ] + } + ], + "team": "T01CWUYP5AR", + "thread_ts": "1699096090.087359", + "reply_count": 9, + "reply_users_count": 2, + "latest_reply": "1699100482.031569", + "reply_users": [ + "U02S6F54MAB", + "U062Q95A1FG" + ], + "is_locked": false, + "subscribed": false, + "replies": [ + { + "client_msg_id": "2e6f8167-50bf-49cf-85d5-24ae87ae8e42", + "type": "message", + "text": "\nDo you need some more guidance than that?", + "user": "U02S6F54MAB", + "ts": "1699096306.006439", + "blocks": [ + { + "type": "rich_text", + "block_id": "FuLGB", + "elements": [ + { + "type": "rich_text_section", + "elements": [ + { + "type": "link", + "url": "https://openlineage.slack.com/archives/C01CK9T7HKR/p1698315220142929" + }, + { + "type": "text", + "text": "\nDo you need some more guidance than that?" + } + ] + } + ] + } + ], + "team": "T01CWUYP5AR", + "edited": { + "user": "U02S6F54MAB", + "ts": "1699096309.000000" + }, + "attachments": [ + { + "from_url": "https://openlineage.slack.com/archives/C01CK9T7HKR/p1698315220142929", + "ts": "1698315220.142929", + "author_id": "U062Q95A1FG", + "channel_id": "C01CK9T7HKR", + "channel_team": "T01CWUYP5AR", + "is_msg_unfurl": true, + "is_thread_root_unfurl": true, + "message_blocks": [ + { + "team": "T01CWUYP5AR", + "channel": "C01CK9T7HKR", + "ts": "1698315220.142929", + "message": { + "blocks": [ + { + "type": "rich_text", + "block_id": "V6ApU", + "elements": [ + { + "type": "rich_text_section", + "elements": [ + { + "type": "text", + "text": "Hi I want to customise the events which comes from Openlineage spark . Can some one give some information" + } + ] + } + ] + } + ] + } + } + ], + "id": 1, + "original_url": "https://openlineage.slack.com/archives/C01CK9T7HKR/p1698315220142929", + "fallback": "[October 26th, 2023 3:13 AM] n.priya88: Hi I want to customise the events which comes from Openlineage spark . Can some one give some information", + "text": "Hi I want to customise the events which comes from Openlineage spark . Can some one give some information", + "author_name": "priya narayana", + "author_link": "https://openlineage.slack.com/team/U062Q95A1FG", + "author_icon": "https://avatars.slack-edge.com/2023-10-26/6084416738247_2017b9ef79397fadc4f2_48.png", + "author_subname": "priya narayana", + "mrkdwn_in": [ + "text" + ], + "footer": "Thread in Slack Conversation" + } + ], + "thread_ts": "1699096090.087359", + "parent_user_id": "U062Q95A1FG" + }, + { + "client_msg_id": "058fb0f2-eaba-45ba-b3a3-3ec93d23ab63", + "type": "message", + "text": "yes", + "user": "U062Q95A1FG", + "ts": "1699096427.399719", + "blocks": [ + { + "type": "rich_text", + "block_id": "IB8ze", + "elements": [ + { + "type": "rich_text_section", + "elements": [ + { + "type": "text", + "text": "yes" + } + ] + } + ] + } + ], + "team": "T01CWUYP5AR", + "thread_ts": "1699096090.087359", + "parent_user_id": "U062Q95A1FG" + }, + { + "client_msg_id": "b32051d2-c4ec-4a36-8a2e-ab7781fc2932", + "type": "message", + "text": "It seems pretty extensively described, what kind of help do you need?", + "user": "U02S6F54MAB", + "ts": "1699096521.602359", + "blocks": [ + { + "type": "rich_text", + "block_id": "0HfAk", + "elements": [ + { + "type": "rich_text_section", + "elements": [ + { + "type": "text", + "text": "It seems pretty extensively described, what kind of help do you need?" + } + ] + } + ] + } + ], + "team": "T01CWUYP5AR", + "thread_ts": "1699096090.087359", + "parent_user_id": "U062Q95A1FG" + }, + { + "client_msg_id": "0bca5058-05bf-4ec9-a103-6e89561627f0", + "type": "message", + "text": "`io.openlineage.spark.api.OpenLineageEventHandlerFactory` if i use this how will i pass custom listener to my spark submit", + "user": "U062Q95A1FG", + "ts": "1699096573.222189", + "blocks": [ + { + "type": "rich_text", + "block_id": "MUpbi", + "elements": [ + { + "type": "rich_text_section", + "elements": [ + { + "type": "text", + "text": "io.openlineage.spark.api.OpenLineageEventHandlerFactory ", + "style": { + "code": true + } + }, + { + "type": "text", + "text": " if i use this how will i pass custom listener to my spark submit" + } + ] + } + ] + } + ], + "team": "T01CWUYP5AR", + "thread_ts": "1699096090.087359", + "parent_user_id": "U062Q95A1FG" + }, + { + "client_msg_id": "521aa664-fd00-4232-b2b3-27ed5970cd1b", + "type": "message", + "text": "I would like to know how will i customize my events using this . For example: - In \"input\" Facet i want only symlinks name i am not intereseted in anything else", + "user": "U062Q95A1FG", + "ts": "1699096645.101619", + "blocks": [ + { + "type": "rich_text", + "block_id": "o62J5", + "elements": [ + { + "type": "rich_text_section", + "elements": [ + { + "type": "text", + "text": "I would like to know how will i customize my events using this . For example: - In \"input\" Facet i want only symlinks name i am not intereseted in anything else" + } + ] + } + ] + } + ], + "team": "T01CWUYP5AR", + "thread_ts": "1699096090.087359", + "parent_user_id": "U062Q95A1FG" + }, + { + "client_msg_id": "27680813-782a-438d-882f-e954856535f3", + "type": "message", + "text": "can you please provide some guidance", + "user": "U062Q95A1FG", + "ts": "1699096652.525109", + "blocks": [ + { + "type": "rich_text", + "block_id": "WuJp0", + "elements": [ + { + "type": "rich_text_section", + "elements": [ + { + "type": "text", + "text": "can you please provide some guidance" + } + ] + } + ] + } + ], + "team": "T01CWUYP5AR", + "thread_ts": "1699096090.087359", + "parent_user_id": "U062Q95A1FG" + }, + { + "client_msg_id": "4cb5c3ae-a67c-40a1-99c4-466f2c7e9574", + "type": "message", + "text": "<@U02S6F54MAB> this is the doubt i have", + "user": "U062Q95A1FG", + "ts": "1699096716.604929", + "blocks": [ + { + "type": "rich_text", + "block_id": "jY65x", + "elements": [ + { + "type": "rich_text_section", + "elements": [ + { + "type": "user", + "user_id": "U02S6F54MAB" + }, + { + "type": "text", + "text": " this is the doubt i have" + } + ] + } + ] + } + ], + "team": "T01CWUYP5AR", + "thread_ts": "1699096090.087359", + "parent_user_id": "U062Q95A1FG" + }, + { + "client_msg_id": "af8afd49-ca6c-42d7-874e-8ef0125d4899", + "type": "message", + "text": "Some one who did spark integration throw some light", + "user": "U062Q95A1FG", + "ts": "1699100245.653179", + "blocks": [ + { + "type": "rich_text", + "block_id": "ebM8j", + "elements": [ + { + "type": "rich_text_section", + "elements": [ + { + "type": "text", + "text": "Some one who did spark integration throw some light" + } + ] + } + ] + } + ], + "team": "T01CWUYP5AR", + "thread_ts": "1699096090.087359", + "parent_user_id": "U062Q95A1FG" + }, + { + "client_msg_id": "d244a1d7-3318-4011-9f53-f849f0c3bbbc", + "type": "message", + "text": "it's weekend for most of us so you probably need to wait until Monday for precise answers", + "user": "U02S6F54MAB", + "ts": "1699100482.031569", + "blocks": [ + { + "type": "rich_text", + "block_id": "sIFCY", + "elements": [ + { + "type": "rich_text_section", + "elements": [ + { + "type": "text", + "text": "it's weekend for most of us so you probably need to wait until Monday for precise answers" + } + ] + } + ] + } + ], + "team": "T01CWUYP5AR", + "thread_ts": "1699096090.087359", + "parent_user_id": "U062Q95A1FG" + } + ] + }, + { + "client_msg_id": "d300023e-b76f-488e-9f6d-9d1a7e7821c9", + "type": "message", + "text": "\nThis month’s TSC meeting (open to all) is next Thursday the 9th at 10am PT. On the agenda:\n• announcements\n• recent releases\n• recent additions to the Flink integration by <@U05QA2D1XNV> \n• recent additions to the Spark integration by <@U02MK6YNAQ5> \n• updates on proposals by <@U01DCLP0GU9> \n• discussion topics\n• open discussion\nMore info and the meeting link can be found on the . All are welcome! Do you have a discussion topic, use case or integration you’d like to demo? DM me to be added to the agenda.", + "user": "U02LXF3HUN7", + "ts": "1699027207.361229", + "blocks": [ + { + "type": "rich_text", + "block_id": "VnOMq", + "elements": [ + { + "type": "rich_text_section", + "elements": [ + { + "type": "broadcast", + "range": "channel" + }, + { + "type": "text", + "text": "\nThis month’s TSC meeting (open to all) is next Thursday the 9th at 10am PT. On the agenda:\n" + } + ] + }, + { + "type": "rich_text_list", + "elements": [ + { + "type": "rich_text_section", + "elements": [ + { + "type": "text", + "text": "announcements" + } + ] + }, + { + "type": "rich_text_section", + "elements": [ + { + "type": "text", + "text": "recent releases" + } + ] + }, + { + "type": "rich_text_section", + "elements": [ + { + "type": "text", + "text": "recent additions to the Flink integration by " + }, + { + "type": "user", + "user_id": "U05QA2D1XNV" + }, + { + "type": "text", + "text": " " + } + ] + }, + { + "type": "rich_text_section", + "elements": [ + { + "type": "text", + "text": "recent additions to the Spark integration by " + }, + { + "type": "user", + "user_id": "U02MK6YNAQ5" + }, + { + "type": "text", + "text": " " + } + ] + }, + { + "type": "rich_text_section", + "elements": [ + { + "type": "text", + "text": "updates on proposals by " + }, + { + "type": "user", + "user_id": "U01DCLP0GU9" + }, + { + "type": "text", + "text": " " + } + ] + }, + { + "type": "rich_text_section", + "elements": [ + { + "type": "text", + "text": "discussion topics" + } + ] + }, + { + "type": "rich_text_section", + "elements": [ + { + "type": "text", + "text": "open discussion" + } + ] + } + ], + "style": "bullet", + "indent": 0, + "border": 0 + }, + { + "type": "rich_text_section", + "elements": [ + { + "type": "text", + "text": "More info and the meeting link can be found on the " + }, + { + "type": "link", + "url": "https://openlineage.io/meetings/", + "text": "website" + }, + { + "type": "text", + "text": ". All are welcome! Do you have a discussion topic, use case or integration you’d like to demo? DM me to be added to the agenda." + } + ] + } + ] + } + ], + "team": "T01CWUYP5AR", + "attachments": [ + { + "from_url": "https://openlineage.io/meetings/", + "service_icon": "https://openlineage.io/img/favicon.ico", + "id": 1, + "original_url": "https://openlineage.io/meetings/", + "fallback": "TSC Meetings | OpenLineage", + "text": "The OpenLineage Technical Steering Committee meets monthly, and is open to all.", + "title": "TSC Meetings | OpenLineage", + "title_link": "https://openlineage.io/meetings/", + "service_name": "openlineage.io" + } + ], + "reactions": [ + { + "name": "+1", + "users": [ + "U062WLFMRTP" + ], + "count": 1 + } + ] + }, + { + "client_msg_id": "240610ee-fc3d-4e71-9f70-731cd88495a6", + "type": "message", + "text": "actually, it shows up in one of the RUNNING now... behavior is consistent between 11.3 and 13.3, thanks for fixing this issue", + "user": "U05T8BJD4DU", + "ts": "1698999491.798599", + "blocks": [ + { + "type": "rich_text", + "block_id": "q6sfC", + "elements": [ + { + "type": "rich_text_section", + "elements": [ + { + "type": "text", + "text": "actually, it shows up in one of the RUNNING now... behavior is consistent between 11.3 and 13.3, thanks for fixing this issue" + } + ] + } + ] + } + ], + "team": "T01CWUYP5AR", + "thread_ts": "1698999491.798599", + "reply_count": 7, + "reply_users_count": 2, + "latest_reply": "1699471566.197189", + "reply_users": [ + "U05T8BJD4DU", + "U05TU0U224A" + ], + "is_locked": false, + "subscribed": false, + "reactions": [ + { + "name": "+1", + "users": [ + "U02MK6YNAQ5" + ], + "count": 1 + } + ], + "replies": [ + { + "client_msg_id": "464bebd8-1178-4639-88a8-8cac9334395b", + "type": "message", + "text": "<@U02MK6YNAQ5> looks like I need to bring bad news.. 13.3 is fixed for specific scenarios, but 11.3 is still reading output as dbfs.. there are scenarios that it's not producing input and output like:\n\ncreate table table using delta as\nlocation 'abfss://....'\nSelect * from parquet.`abfss://....'", + "user": "U05T8BJD4DU", + "ts": "1699127062.015709", + "blocks": [ + { + "type": "rich_text", + "block_id": "1hVmH", + "elements": [ + { + "type": "rich_text_section", + "elements": [ + { + "type": "user", + "user_id": "U02MK6YNAQ5" + }, + { + "type": "text", + "text": " looks like I need to bring bad news.. 13.3 is fixed for specific scenarios, but 11.3 is still reading output as dbfs.. there are scenarios that it's not producing input and output like:\n\ncreate table table using delta as\nlocation 'abfss://....'\nSelect * from parquet.`abfss://....'" + } + ] + } + ] + } + ], + "team": "T01CWUYP5AR", + "thread_ts": "1698999491.798599", + "parent_user_id": "U05T8BJD4DU" + }, + { + "client_msg_id": "6ba957bb-7e64-43f7-be80-de4d9c568138", + "type": "message", + "text": "Will test more and ope issues", + "user": "U05T8BJD4DU", + "ts": "1699127071.990639", + "blocks": [ + { + "type": "rich_text", + "block_id": "EiQMS", + "elements": [ + { + "type": "rich_text_section", + "elements": [ + { + "type": "text", + "text": "Will test more and ope issues" + } + ] + } + ] + } + ], + "team": "T01CWUYP5AR", + "thread_ts": "1698999491.798599", + "parent_user_id": "U05T8BJD4DU" + }, + { + "client_msg_id": "83d639e6-a07c-4463-9797-1f6e6abc8381", + "type": "message", + "text": "<@U05T8BJD4DU>how did you manage the get the environment attribute. it's not showing up to me at all. I've tried databricks abut also tried a local instance of spark.", + "user": "U05TU0U224A", + "ts": "1699266873.918299", + "blocks": [ + { + "type": "rich_text", + "block_id": "0GwWq", + "elements": [ + { + "type": "rich_text_section", + "elements": [ + { + "type": "user", + "user_id": "U05T8BJD4DU" + }, + { + "type": "text", + "text": "how did you manage the get the environment attribute. it's not showing up to me at all. I've tried databricks abut also tried a local instance of spark." + } + ] + } + ] + } + ], + "team": "T01CWUYP5AR", + "thread_ts": "1698999491.798599", + "parent_user_id": "U05T8BJD4DU" + }, + { + "client_msg_id": "0c036060-f060-45fb-9e10-84a1d263d415", + "type": "message", + "text": "<@U05TU0U224A> its showing up in one of the RUNNING events, not in the START event anymore", + "user": "U05T8BJD4DU", + "ts": "1699399922.001119", + "blocks": [ + { + "type": "rich_text", + "block_id": "SQy71", + "elements": [ + { + "type": "rich_text_section", + "elements": [ + { + "type": "user", + "user_id": "U05TU0U224A" + }, + { + "type": "text", + "text": " its showing up in one of the RUNNING events, not in the START event anymore" + } + ] + } + ] + } + ], + "team": "T01CWUYP5AR", + "thread_ts": "1698999491.798599", + "parent_user_id": "U05T8BJD4DU" + }, + { + "client_msg_id": "be6884cc-251f-4bed-8e54-cc990b344860", + "type": "message", + "text": "I never had a running event :melting_face: Am I filtering something?", + "user": "U05TU0U224A", + "ts": "1699430672.649399", + "blocks": [ + { + "type": "rich_text", + "block_id": "1HOGl", + "elements": [ + { + "type": "rich_text_section", + "elements": [ + { + "type": "text", + "text": "I never had a running event " + }, + { + "type": "emoji", + "name": "melting_face", + "unicode": "1fae0" + }, + { + "type": "text", + "text": " Am I filtering something?" + } + ] + } + ] + } + ], + "team": "T01CWUYP5AR", + "thread_ts": "1698999491.798599", + "parent_user_id": "U05T8BJD4DU" + }, + { + "client_msg_id": "c590f6a7-e029-4118-b6e7-66f1c6a2c4cf", + "type": "message", + "text": "Umm.. ok show me your code, will try on my end", + "user": "U05T8BJD4DU", + "ts": "1699466606.346779", + "blocks": [ + { + "type": "rich_text", + "block_id": "4l7gs", + "elements": [ + { + "type": "rich_text_section", + "elements": [ + { + "type": "text", + "text": "Umm.. ok show me your code, will try on my end" + } + ] + } + ] + } + ], + "team": "T01CWUYP5AR", + "thread_ts": "1698999491.798599", + "parent_user_id": "U05T8BJD4DU" + }, + { + "client_msg_id": "72b08038-c170-4b56-b2dd-1db98d7434e9", + "type": "message", + "text": "<@U02MK6YNAQ5> <@U05TU0U224A> actually if you are using UC-enabled cluster, you won't get any RUNNING events", + "user": "U05T8BJD4DU", + "ts": "1699471566.197189", + "blocks": [ + { + "type": "rich_text", + "block_id": "JozE0", + "elements": [ + { + "type": "rich_text_section", + "elements": [ + { + "type": "user", + "user_id": "U02MK6YNAQ5" + }, + { + "type": "text", + "text": " " + }, + { + "type": "user", + "user_id": "U05TU0U224A" + }, + { + "type": "text", + "text": " actually if you are using UC-enabled cluster, you won't get any RUNNING events" + } + ] + } + ] + } + ], + "team": "T01CWUYP5AR", + "thread_ts": "1698999491.798599", + "parent_user_id": "U05T8BJD4DU" + } + ] + }, + { + "type": "message", + "text": "<@U02MK6YNAQ5> I tested 1.5.0, it works great now, but the environment facets is gone in START... which I very much want it.. any thoughts?", + "files": [ + { + "id": "F063M06MWBZ", + "created": 1698950948, + "timestamp": 1698950948, + "name": "data (10).json", + "title": "data (10).json", + "mimetype": "text/plain", + "filetype": "json", + "pretty_type": "JSON", + "user": "U05T8BJD4DU", + "user_team": "T01CWUYP5AR", + "editable": true, + "size": 24081, + "mode": "snippet", + "is_external": false, + "external_type": "", + "is_public": true, + "public_url_shared": false, + "display_as_bot": false, + "username": "", + "url_private": "https://files.slack.com/files-pri/T01CWUYP5AR-F063M06MWBZ/data__10_.json", + "url_private_download": "https://files.slack.com/files-pri/T01CWUYP5AR-F063M06MWBZ/download/data__10_.json", + "permalink": "https://openlineage.slack.com/files/U05T8BJD4DU/F063M06MWBZ/data__10_.json", + "permalink_public": "https://slack-files.com/T01CWUYP5AR-F063M06MWBZ-6b6cda4177", + "edit_link": "https://openlineage.slack.com/files/U05T8BJD4DU/F063M06MWBZ/data__10_.json/edit", + "preview": "{\r\n \"eventTime\":\"2023-11-02T18:42:00.619Z\",\r\n \"producer\":\"https://github.com/OpenLineage/OpenLineage/tree/1.5.0/integration/spark\",\r\n \"schemaURL\":\"https://openlineage.io/spec/2-0-2/OpenLineage.json#/$defs/RunEvent\",\r\n \"eventType\":\"START\",\r", + "preview_highlight": "
\n
\n
{
\n
   "eventTime":"2023-11-02T18:42:00.619Z",
\n
   "producer":"https://github.com/OpenLineage/OpenLineage/tree/1.5.0/integration/spark",
\n
   "schemaURL":"https://openlineage.io/spec/2-0-2/OpenLineage.json#/$defs/RunEvent",
\n
   "eventType":"START",
\n
\n
\n", + "lines": 531, + "lines_more": 526, + "preview_is_truncated": true, + "is_starred": false, + "has_rich_preview": false, + "file_access": "visible" + } + ], + "upload": false, + "user": "U05T8BJD4DU", + "display_as_bot": false, + "ts": "1698950958.157459", + "blocks": [ + { + "type": "rich_text", + "block_id": "v2Gft", + "elements": [ + { + "type": "rich_text_section", + "elements": [ + { + "type": "user", + "user_id": "U02MK6YNAQ5" + }, + { + "type": "text", + "text": " I tested 1.5.0, it works great now, but the environment facets is gone in START... which I very much want it.. any thoughts?" + } + ] + } + ] + } + ], + "client_msg_id": "9b3348ec-8f79-4822-ae9c-44c87120828f" + }, + { + "client_msg_id": "dd052197-8cbb-4f3c-bf1b-d22cc8d9fb98", + "type": "message", + "text": "\nWe released OpenLineage 1.5.0, including:\n• by <@U05QA2D1XNV> \n• by <@U02MK6YNAQ5> \n• `rdd``toDF` by <@U02MK6YNAQ5> \n• by <@U02S6F54MAB> \n• by <@U02S6F54MAB> \n• bug fixes, tests, infra fixes, doc changes, and more.\nThanks to all the contributors, including new contributor <@U05VDHJJ9T7>!\n*Release:* \n*Changelog:* \n*Commit history:* \n*Maven:* \n*PyPI:* ", + "user": "U02LXF3HUN7", + "ts": "1698940800.306129", + "blocks": [ + { + "type": "rich_text", + "block_id": "Me1wK", + "elements": [ + { + "type": "rich_text_section", + "elements": [ + { + "type": "broadcast", + "range": "channel" + }, + { + "type": "text", + "text": "\nWe released OpenLineage 1.5.0, including:\n" + } + ] + }, + { + "type": "rich_text_list", + "elements": [ + { + "type": "rich_text_section", + "elements": [ + { + "type": "link", + "url": "https://github.com/OpenLineage/OpenLineage/pull/2175", + "text": "support for Cassandra Connectors lineage in the Flink integration" + }, + { + "type": "text", + "text": " by " + }, + { + "type": "user", + "user_id": "U05QA2D1XNV" + }, + { + "type": "text", + "text": " " + } + ] + }, + { + "type": "rich_text_section", + "elements": [ + { + "type": "link", + "url": "https://github.com/OpenLineage/OpenLineage/pull/2185", + "text": "support for Databricks Runtime 13.3 in the Spark integration" + }, + { + "type": "text", + "text": " by " + }, + { + "type": "user", + "user_id": "U02MK6YNAQ5" + }, + { + "type": "text", + "text": " " + } + ] + }, + { + "type": "rich_text_section", + "elements": [ + { + "type": "link", + "url": "https://github.com/OpenLineage/OpenLineage/pull/2188", + "text": "support for " + }, + { + "type": "text", + "text": "rdd", + "style": { + "code": true + } + }, + { + "type": "link", + "url": "https://github.com/OpenLineage/OpenLineage/pull/2188", + "text": " and " + }, + { + "type": "text", + "text": "toDF", + "style": { + "code": true + } + }, + { + "type": "link", + "url": "https://github.com/OpenLineage/OpenLineage/pull/2188", + "text": " operations from the Spark Scala API in Spark" + }, + { + "type": "text", + "text": " by " + }, + { + "type": "user", + "user_id": "U02MK6YNAQ5" + }, + { + "type": "text", + "text": " " + } + ] + }, + { + "type": "rich_text_section", + "elements": [ + { + "type": "link", + "url": "https://github.com/OpenLineage/OpenLineage/pull/2107", + "text": "lowered requirements for attrs and requests packages in the Airflow integration" + }, + { + "type": "text", + "text": " by " + }, + { + "type": "user", + "user_id": "U02S6F54MAB" + }, + { + "type": "text", + "text": " " + } + ] + }, + { + "type": "rich_text_section", + "elements": [ + { + "type": "link", + "url": "https://github.com/OpenLineage/OpenLineage/pull/2221", + "text": "lazy rendering of yaml configs in the dbt integration" + }, + { + "type": "text", + "text": " by " + }, + { + "type": "user", + "user_id": "U02S6F54MAB" + }, + { + "type": "text", + "text": " " + } + ] + }, + { + "type": "rich_text_section", + "elements": [ + { + "type": "text", + "text": "bug fixes, tests, infra fixes, doc changes, and more." + } + ] + } + ], + "style": "bullet", + "indent": 0, + "border": 0 + }, + { + "type": "rich_text_section", + "elements": [ + { + "type": "text", + "text": "Thanks to all the contributors, including new contributor " + }, + { + "type": "user", + "user_id": "U05VDHJJ9T7" + }, + { + "type": "text", + "text": "!\n" + }, + { + "type": "text", + "text": "Release:", + "style": { + "bold": true + } + }, + { + "type": "text", + "text": " " + }, + { + "type": "link", + "url": "https://github.com/OpenLineage/OpenLineage/releases/tag/1.5.0" + }, + { + "type": "text", + "text": "\n" + }, + { + "type": "text", + "text": "Changelog: ", + "style": { + "bold": true + } + }, + { + "type": "link", + "url": "https://github.com/OpenLineage/OpenLineage/blob/main/CHANGELOG.md" + }, + { + "type": "text", + "text": "\n" + }, + { + "type": "text", + "text": "Commit history:", + "style": { + "bold": true + } + }, + { + "type": "text", + "text": " " + }, + { + "type": "link", + "url": "https://github.com/OpenLineage/OpenLineage/compare/1.4.1...1.5.0" + }, + { + "type": "text", + "text": "\n" + }, + { + "type": "text", + "text": "Maven:", + "style": { + "bold": true + } + }, + { + "type": "text", + "text": " " + }, + { + "type": "link", + "url": "https://oss.sonatype.org/#nexus-search;quick~openlineage" + }, + { + "type": "text", + "text": "\n" + }, + { + "type": "text", + "text": "PyPI:", + "style": { + "bold": true + } + }, + { + "type": "text", + "text": " " + }, + { + "type": "link", + "url": "https://pypi.org/project/openlineage-python/" + } + ] + } + ] + } + ], + "team": "T01CWUYP5AR", + "reactions": [ + { + "name": "+1", + "users": [ + "U05T8BJD4DU", + "U05VDHJJ9T7", + "U053LCT71BQ", + "U01HVNU6A4C", + "U05SMTVPPL3" + ], + "count": 5 + }, + { + "name": "rocket", + "users": [ + "U055N2GRT4P" + ], + "count": 1 + } + ] + }, + { + "client_msg_id": "d52119da-eee6-494c-a28e-aab20a0f55a5", + "type": "message", + "text": "I am looking to send OpenLineage events to an AWS API Gateway endpoint from an AWS MWAA instance. The problem is that all requests to AWS services need to be signed with SigV4, and using API Gateway with IAM authentication would require requests to API Gateway be signed with SigV4. Would the best way to do so be to just modify the python client HTTP transport to include a new config option for signing emitted OpenLineage events with SigV4? Are there any alternatives?", + "user": "U063YP6UJJ0", + "ts": "1698885038.172079", + "blocks": [ + { + "type": "rich_text", + "block_id": "uukJK", + "elements": [ + { + "type": "rich_text_section", + "elements": [ + { + "type": "text", + "text": "I am looking to send OpenLineage events to an AWS API Gateway endpoint from an AWS MWAA instance. The problem is that all requests to AWS services need to be signed with SigV4, and using API Gateway with IAM authentication would require requests to API Gateway be signed with SigV4. Would the best way to do so be to just modify the python client HTTP transport to include a new config option for signing emitted OpenLineage events with SigV4? Are there any alternatives?" + } + ] + } + ] + } + ], + "team": "T01CWUYP5AR", + "edited": { + "user": "U063YP6UJJ0", + "ts": "1698885095.000000" + }, + "thread_ts": "1698885038.172079", + "reply_count": 7, + "reply_users_count": 2, + "latest_reply": "1699030375.287509", + "reply_users": [ + "U02S6F54MAB", + "U063YP6UJJ0" + ], + "is_locked": false, + "subscribed": false, + "replies": [ + { + "client_msg_id": "cac14e88-5767-4eee-81fc-7978de1c6b8e", + "type": "message", + "text": "there’s actually an issue for that:\n\n\nbut the way to do this is imho to create new custom transport (it might inherit from HTTP transport) and register it in transport factory", + "user": "U02S6F54MAB", + "ts": "1698907310.848799", + "blocks": [ + { + "type": "rich_text", + "block_id": "xfpF9", + "elements": [ + { + "type": "rich_text_section", + "elements": [ + { + "type": "text", + "text": "there’s actually an issue for that:\n" + }, + { + "type": "link", + "url": "https://github.com/OpenLineage/OpenLineage/issues/2189" + }, + { + "type": "text", + "text": "\n\nbut the way to do this is imho to create new custom transport (it might inherit from HTTP transport) and register it in transport factory" + } + ] + } + ] + } + ], + "team": "T01CWUYP5AR", + "thread_ts": "1698885038.172079", + "parent_user_id": "U063YP6UJJ0" + }, + { + "client_msg_id": "0d988f4f-3290-485d-ae49-627495942fee", + "type": "message", + "text": "I am thinking of just modifying the HTTP transport and using requests.auth.AuthBase to create different auth methods instead of a TokenProvider class\n\nClasses which subclass requests.auth.AuthBase can also just directly be given to the requests call in the auth parameter", + "user": "U063YP6UJJ0", + "ts": "1698944705.537979", + "blocks": [ + { + "type": "rich_text", + "block_id": "IxFOZ", + "elements": [ + { + "type": "rich_text_section", + "elements": [ + { + "type": "text", + "text": "I am thinking of just modifying the HTTP transport and using requests.auth.AuthBase to create different auth methods instead of a TokenProvider class\n\nClasses which subclass requests.auth.AuthBase can also just directly be given to the requests call in the auth parameter" + } + ] + } + ] + } + ], + "team": "T01CWUYP5AR", + "edited": { + "user": "U063YP6UJJ0", + "ts": "1698944761.000000" + }, + "thread_ts": "1698885038.172079", + "parent_user_id": "U063YP6UJJ0", + "reactions": [ + { + "name": "+1", + "users": [ + "U02S6F54MAB" + ], + "count": 1 + } + ] + }, + { + "client_msg_id": "795deede-6d64-4abb-98e4-0c703f2bbeba", + "type": "message", + "text": "would you like to contribute? :slightly_smiling_face:", + "user": "U02S6F54MAB", + "ts": "1698950424.250389", + "blocks": [ + { + "type": "rich_text", + "block_id": "PVFJT", + "elements": [ + { + "type": "rich_text_section", + "elements": [ + { + "type": "text", + "text": "would you like to contribute? " + }, + { + "type": "emoji", + "name": "slightly_smiling_face", + "unicode": "1f642" + } + ] + } + ] + } + ], + "team": "T01CWUYP5AR", + "thread_ts": "1698885038.172079", + "parent_user_id": "U063YP6UJJ0" + }, + { + "client_msg_id": "29e7aa89-d809-4605-a27d-e56951ddbef4", + "type": "message", + "text": "I was about to contribute, but I actually just realized that there is an existing way to provide a custom transport that would solve form y use case. My only question is how do I register this custom transport in my MWAA environment? Can I provide the custom transport as an Airflow plugin and then specify the class in the Openlineage.yml config? Will it automatically pick it up?", + "user": "U063YP6UJJ0", + "ts": "1698950585.337849", + "blocks": [ + { + "type": "rich_text", + "block_id": "zxE+U", + "elements": [ + { + "type": "rich_text_section", + "elements": [ + { + "type": "text", + "text": "I was about to contribute, but I actually just realized that there is an existing way to provide a custom transport that would solve form y use case. My only question is how do I register this custom transport in my MWAA environment? Can I provide the custom transport as an Airflow plugin and then specify the class in the Openlineage.yml config? Will it automatically pick it up?" + } + ] + } + ] + } + ], + "team": "T01CWUYP5AR", + "thread_ts": "1698885038.172079", + "parent_user_id": "U063YP6UJJ0" + }, + { + "client_msg_id": "67a9dc7f-69c9-4e38-92d6-221ed3e5b652", + "type": "message", + "text": "although I did not test this in MWAA but locally only: I’ve created Airflow plugin that in `__init__.py` has defined (or imported) following code:\n```from openlineage.client.transport import register_transport, Transport, Config\n\n\n@register_transport\nclass FakeTransport(Transport):\n kind = \"fake\"\n config = Config\n\n def __init__(self, config: Config) -> None:\n print(config)\n\n def emit(self, event) -> None:\n print(event)```\nsetting `AIRFLOW__OPENLINEAGE__TRANSPORT='{\"type\": \"fake\"}'` does take effect and I can see output in Airflow logs", + "user": "U02S6F54MAB", + "ts": "1698954356.104239", + "blocks": [ + { + "type": "rich_text", + "block_id": "a3/UA", + "elements": [ + { + "type": "rich_text_section", + "elements": [ + { + "type": "text", + "text": "although I did not test this in MWAA but locally only: I’ve created Airflow plugin that in " + }, + { + "type": "text", + "text": "__init__.py", + "style": { + "code": true + } + }, + { + "type": "text", + "text": " has defined (or imported) following code:\n" + } + ] + }, + { + "type": "rich_text_preformatted", + "elements": [ + { + "type": "text", + "text": "from openlineage.client.transport import register_transport, Transport, Config\n\n\n@register_transport\nclass FakeTransport(Transport):\n kind = \"fake\"\n config = Config\n\n def __init__(self, config: Config) -> None:\n print(config)\n\n def emit(self, event) -> None:\n print(event)" + } + ], + "border": 0 + }, + { + "type": "rich_text_section", + "elements": [ + { + "type": "text", + "text": "setting " + }, + { + "type": "text", + "text": "AIRFLOW__OPENLINEAGE__TRANSPORT='{\"type\": \"fake\"}'", + "style": { + "code": true + } + }, + { + "type": "text", + "text": " does take effect and I can see output in Airflow logs" + } + ] + } + ] + } + ], + "team": "T01CWUYP5AR", + "edited": { + "user": "U02S6F54MAB", + "ts": "1698954364.000000" + }, + "thread_ts": "1698885038.172079", + "parent_user_id": "U063YP6UJJ0" + }, + { + "client_msg_id": "526277f0-8c1d-462a-a361-b66d1e4baa53", + "type": "message", + "text": "in `setup.py` it’s:\n``` ...,\n entry_points={\n 'airflow.plugins': [\n 'custom_transport = custom_transport:CustomTransportPlugin',\n ],\n },\n install_requires=[\"openlineage-python\"]\n)```", + "user": "U02S6F54MAB", + "ts": "1698954465.672369", + "blocks": [ + { + "type": "rich_text", + "block_id": "Bv4qn", + "elements": [ + { + "type": "rich_text_section", + "elements": [ + { + "type": "text", + "text": "in " + }, + { + "type": "text", + "text": "setup.py", + "style": { + "code": true + } + }, + { + "type": "text", + "text": " it’s:\n" + } + ] + }, + { + "type": "rich_text_preformatted", + "elements": [ + { + "type": "text", + "text": " ...,\n entry_points={\n 'airflow.plugins': [\n 'custom_transport = custom_transport:CustomTransportPlugin',\n ],\n },\n install_requires=[\"openlineage-python\"]\n)" + } + ], + "border": 0 + } + ] + } + ], + "team": "T01CWUYP5AR", + "thread_ts": "1698885038.172079", + "parent_user_id": "U063YP6UJJ0" + }, + { + "client_msg_id": "d97a0613-2ab0-423c-81ba-429b592d5351", + "type": "message", + "text": "ok great thanks for following up on this, super helpful", + "user": "U063YP6UJJ0", + "ts": "1699030375.287509", + "blocks": [ + { + "type": "rich_text", + "block_id": "Gsemv", + "elements": [ + { + "type": "rich_text_section", + "elements": [ + { + "type": "text", + "text": "ok great thanks for following up on this, super helpful" + } + ] + } + ] + } + ], + "team": "T01CWUYP5AR", + "thread_ts": "1698885038.172079", + "parent_user_id": "U063YP6UJJ0" + } + ] + }, + { + "client_msg_id": "6f5d63b5-c671-48b3-85df-b49bd907adc0", + "type": "message", + "text": "Hi team :wave: , we’re finding that for our Spark jobs we are almost always getting some junk characters in our dataset names. We’ve pushed the regex filter to its limits and would like to extend the logic of deriving the dataset name in openlineage-spark (currently on `1.4.1`). I seem to recall hearing we could do this by implementing our own `LogicalPlanVisitor` or something along those lines? Is that still the recommended approach and if so would this be possible to implement in Scala vs. Java (scala noob here :simple_smile:)", + "user": "U04AZ7992SU", + "ts": "1698882039.335099", + "blocks": [ + { + "type": "rich_text", + "block_id": "pGDVg", + "elements": [ + { + "type": "rich_text_section", + "elements": [ + { + "type": "text", + "text": "Hi team " + }, + { + "type": "emoji", + "name": "wave", + "unicode": "1f44b" + }, + { + "type": "text", + "text": " , we’re finding that for our Spark jobs we are almost always getting some junk characters in our dataset names. We’ve pushed the regex filter to its limits and would like to extend the logic of deriving the dataset name in openlineage-spark (currently on " + }, + { + "type": "text", + "text": "1.4.1", + "style": { + "code": true + } + }, + { + "type": "text", + "text": "). I seem to recall hearing we could do this by implementing our own " + }, + { + "type": "text", + "text": "LogicalPlanVisitor", + "style": { + "code": true + } + }, + { + "type": "text", + "text": " or something along those lines? Is that still the recommended approach and if so would this be possible to implement in Scala vs. Java (scala noob here " + }, + { + "type": "emoji", + "name": "simple_smile" + }, + { + "type": "text", + "text": ")" + } + ] + } + ] + } + ], + "team": "T01CWUYP5AR", + "thread_ts": "1698882039.335099", + "reply_count": 2, + "reply_users_count": 1, + "latest_reply": "1698910487.005589", + "reply_users": [ + "U02MK6YNAQ5" + ], + "is_locked": false, + "subscribed": false, + "replies": [ + { + "client_msg_id": "a2380d59-d730-4d2c-95c7-f79c66e609b0", + "type": "message", + "text": "Hi John, we're always happy to help with the contribution.\n\nOne of the possible solutions to this would be to do that just in `openlineage-java` client:\n• introduce config entry like `normalizeDatasetNameToAscii` : `enabled/disabled`\n• modify `DatasetIdentifier` class to contain static member `boolean normalizeDatasetNameToAscii` and normalize dataset name according to this setting\n• additionally, you would need to add config entry in `io.openlineage.client.OpenLineageYaml` and make sure both `loadOpenLineageYaml` methods set `DatasetIdentifier.normalizeDatasetNameToAscii` based on the config\n• document this in the doc\nSo, no Scala nor custom logical plan visitors required.", + "user": "U02MK6YNAQ5", + "ts": "1698910455.477209", + "blocks": [ + { + "type": "rich_text", + "block_id": "gWpRf", + "elements": [ + { + "type": "rich_text_section", + "elements": [ + { + "type": "text", + "text": "Hi John, we're always happy to help with the contribution.\n\nOne of the possible solutions to this would be to do that just in " + }, + { + "type": "text", + "text": "openlineage-java", + "style": { + "code": true + } + }, + { + "type": "text", + "text": " client:\n" + } + ] + }, + { + "type": "rich_text_list", + "elements": [ + { + "type": "rich_text_section", + "elements": [ + { + "type": "text", + "text": "introduce config entry like " + }, + { + "type": "text", + "text": "normalizeDatasetNameToAscii", + "style": { + "code": true + } + }, + { + "type": "text", + "text": " : " + }, + { + "type": "text", + "text": "enabled/disabled", + "style": { + "code": true + } + } + ] + }, + { + "type": "rich_text_section", + "elements": [ + { + "type": "text", + "text": "modify " + }, + { + "type": "text", + "text": "DatasetIdentifier", + "style": { + "code": true + } + }, + { + "type": "text", + "text": " class to contain static member " + }, + { + "type": "text", + "text": "boolean normalizeDatasetNameToAscii", + "style": { + "code": true + } + }, + { + "type": "text", + "text": " and normalize dataset name according to this setting" + } + ] + }, + { + "type": "rich_text_section", + "elements": [ + { + "type": "text", + "text": "additionally, you would need to add config entry in " + }, + { + "type": "text", + "text": "io.openlineage.client.OpenLineageYaml", + "style": { + "code": true + } + }, + { + "type": "text", + "text": " and make sure both " + }, + { + "type": "text", + "text": "loadOpenLineageYaml", + "style": { + "code": true + } + }, + { + "type": "text", + "text": " methods set " + }, + { + "type": "text", + "text": "DatasetIdentifier.normalizeDatasetNameToAscii", + "style": { + "code": true + } + }, + { + "type": "text", + "text": " based on the config" + } + ] + }, + { + "type": "rich_text_section", + "elements": [ + { + "type": "text", + "text": "document this in the doc" + } + ] + } + ], + "style": "bullet", + "indent": 0, + "border": 0 + }, + { + "type": "rich_text_section", + "elements": [ + { + "type": "text", + "text": "\nSo, no Scala nor custom logical plan visitors required." + } + ] + } + ] + } + ], + "team": "T01CWUYP5AR", + "thread_ts": "1698882039.335099", + "parent_user_id": "U04AZ7992SU" + }, + { + "client_msg_id": "08076597-aebc-44fb-bbaa-bc9d3b22e344", + "type": "message", + "text": "", + "user": "U02MK6YNAQ5", + "ts": "1698910487.005589", + "blocks": [ + { + "type": "rich_text", + "block_id": "Cr6n9", + "elements": [ + { + "type": "rich_text_section", + "elements": [ + { + "type": "link", + "url": "https://github.com/OpenLineage/OpenLineage/blob/main/client/java/src/main/java/io/openlineage/client/utils/DatasetIdentifier.java" + } + ] + } + ] + } + ], + "team": "T01CWUYP5AR", + "attachments": [ + { + "id": 1, + "footer_icon": "https://slack.github.com/static/img/favicon-neutral.png", + "color": "24292f", + "bot_id": "B01VA0FB340", + "app_unfurl_url": "https://github.com/OpenLineage/OpenLineage/blob/main/client/java/src/main/java/io/openlineage/client/utils/DatasetIdentifier.java", + "is_app_unfurl": true, + "app_id": "A01BP7R4KNY", + "fallback": "", + "text": "```\n/*\n/* Copyright 2018-2023 contributors to the OpenLineage project\n/* SPDX-License-Identifier: Apache-2.0\n*/\npackage io.openlineage.client.utils;\n\nimport java.util.LinkedList;\nimport java.util.List;\nimport lombok.Value;\n\n@Value\npublic class DatasetIdentifier {\n String name;\n String namespace;\n List symlinks;\n\n public enum SymlinkType {\n TABLE\n };\n\n public DatasetIdentifier(String name, String namespace) {\n this.name = name;\n this.namespace = namespace;\n this.symlinks = new LinkedList<>();\n }\n\n public DatasetIdentifier withSymlink(String name, String namespace, SymlinkType type) {\n symlinks.add(new Symlink(name, namespace, type));\n return this;\n }\n\n public DatasetIdentifier withSymlink(Symlink symlink) {\n symlinks.add(symlink);\n return this;\n }\n\n @Value\n public static class Symlink {\n String name;\n String namespace;\n SymlinkType type;\n }\n}\n\n```", + "title": "", + "footer": "", + "mrkdwn_in": [ + "text" + ] + } + ], + "thread_ts": "1698882039.335099", + "parent_user_id": "U04AZ7992SU", + "reactions": [ + { + "name": "raised_hands", + "users": [ + "U04AZ7992SU" + ], + "count": 1 + } + ] + } + ] + }, + { + "client_msg_id": "fac0c1bd-2166-44fe-92a9-70fb0ae9fb93", + "type": "message", + "text": "\nThe October 2023 issue of is available now! to get in directly in your inbox each month.", + "user": "U02LXF3HUN7", + "ts": "1698859749.531699", + "blocks": [ + { + "type": "rich_text", + "block_id": "n//9G", + "elements": [ + { + "type": "rich_text_section", + "elements": [ + { + "type": "broadcast", + "range": "channel" + }, + { + "type": "text", + "text": "\nThe October 2023 issue of " + }, + { + "type": "link", + "url": "https://mailchi.mp/cea829d27acd/openlineage-news-july-9597657?e=ef0563a7f8", + "text": "OpenLineage News" + }, + { + "type": "text", + "text": " is available now! " + }, + { + "type": "link", + "url": "https://openlineage.us14.list-manage.com/track/click?u=fe7ef7a8dbb32933f30a10466&id=123767f606&e=ef0563a7f8", + "text": "Sign up" + }, + { + "type": "text", + "text": " to get in directly in your inbox each month." + } + ] + } + ] + } + ], + "team": "T01CWUYP5AR", + "attachments": [ + { + "from_url": "https://openlineage.us14.list-manage.com/track/click?u=fe7ef7a8dbb32933f30a10466&id=123767f606&e=ef0563a7f8", + "id": 1, + "original_url": "https://openlineage.us14.list-manage.com/track/click?u=fe7ef7a8dbb32933f30a10466&id=123767f606&e=ef0563a7f8", + "fallback": "OpenLineage Project", + "text": "OpenLineage Project Email Forms", + "title": "OpenLineage Project", + "title_link": "https://openlineage.us14.list-manage.com/track/click?u=fe7ef7a8dbb32933f30a10466&id=123767f606&e=ef0563a7f8", + "service_name": "apache.us14.list-manage.com" + } + ], + "reactions": [ + { + "name": "+1", + "users": [ + "U01HVNU6A4C", + "U062WLFMRTP" + ], + "count": 2 + }, + { + "name": "tada", + "users": [ + "U055N2GRT4P" + ], + "count": 1 + } + ] + }, + { + "client_msg_id": "8f19ced9-362d-47f7-837d-a47df88bbcae", + "type": "message", + "text": "\nI’m opening a vote to release OpenLineage 1.5.0, including:\n• support for Cassandra Connectors lineage in the Flink integration\n• support for Databricks Runtime 13.3 in the Spark integration\n• support for `rdd` and `toDF` operations from the Spark Scala API in Spark\n• lowered requirements for attrs and requests packages in the Airflow integration\n• lazy rendering of yaml configs in the dbt integration\n• bug fixes, tests, infra fixes, doc changes, and more.\nThree +1s from committers will authorize an immediate release.", + "user": "U02LXF3HUN7", + "ts": "1698852883.658009", + "blocks": [ + { + "type": "rich_text", + "block_id": "dgGRj", + "elements": [ + { + "type": "rich_text_section", + "elements": [ + { + "type": "broadcast", + "range": "channel" + }, + { + "type": "text", + "text": "\nI’m opening a vote to release OpenLineage 1.5.0, including:\n" + } + ] + }, + { + "type": "rich_text_list", + "elements": [ + { + "type": "rich_text_section", + "elements": [ + { + "type": "text", + "text": "support for Cassandra Connectors lineage in the Flink integration" + } + ] + }, + { + "type": "rich_text_section", + "elements": [ + { + "type": "text", + "text": "support for Databricks Runtime 13.3 in the Spark integration" + } + ] + }, + { + "type": "rich_text_section", + "elements": [ + { + "type": "text", + "text": "support for " + }, + { + "type": "text", + "text": "rdd", + "style": { + "code": true + } + }, + { + "type": "text", + "text": " and " + }, + { + "type": "text", + "text": "toDF", + "style": { + "code": true + } + }, + { + "type": "text", + "text": " operations from the Spark Scala API in Spark" + } + ] + }, + { + "type": "rich_text_section", + "elements": [ + { + "type": "text", + "text": "lowered requirements for attrs and requests packages in the Airflow integration" + } + ] + }, + { + "type": "rich_text_section", + "elements": [ + { + "type": "text", + "text": "lazy rendering of yaml configs in the dbt integration" + } + ] + }, + { + "type": "rich_text_section", + "elements": [ + { + "type": "text", + "text": "bug fixes, tests, infra fixes, doc changes, and more." + } + ] + } + ], + "style": "bullet", + "indent": 0, + "border": 0 + }, + { + "type": "rich_text_section", + "elements": [ + { + "type": "text", + "text": "Three +1s from committers will authorize an immediate release." + } + ] + } + ] + } + ], + "team": "T01CWUYP5AR", + "thread_ts": "1698852883.658009", + "reply_count": 1, + "reply_users_count": 1, + "latest_reply": "1698916318.596059", + "reply_users": [ + "U02LXF3HUN7" + ], + "is_locked": false, + "subscribed": true, + "last_read": "1698916318.596059", + "reactions": [ + { + "name": "heavy_plus_sign", + "users": [ + "U02S6F54MAB", + "U028NM4NR1Q", + "U05HBLE7YPL", + "U01DCMDFHBK", + "U02MK6YNAQ5", + "U01DCLP0GU9" + ], + "count": 6 + }, + { + "name": "+1", + "users": [ + "U05T8BJD4DU" + ], + "count": 1 + }, + { + "name": "rocket", + "users": [ + "U032GGJESD6", + "U055N2GRT4P" + ], + "count": 2 + } + ], + "replies": [ + { + "client_msg_id": "7aaef1f1-63eb-4a4d-8eaf-4e80d905ad4d", + "type": "message", + "text": "Thanks, all. The release is authorized and will be initiated within 2 business days.", + "user": "U02LXF3HUN7", + "ts": "1698916318.596059", + "blocks": [ + { + "type": "rich_text", + "block_id": "mCijO", + "elements": [ + { + "type": "rich_text_section", + "elements": [ + { + "type": "text", + "text": "Thanks, all. The release is authorized and will be initiated within 2 business days." + } + ] + } + ] + } + ], + "team": "T01CWUYP5AR", + "thread_ts": "1698852883.658009", + "parent_user_id": "U02LXF3HUN7" + } + ] + }, + { + "type": "message", + "subtype": "thread_broadcast", + "text": "one question if someone is around - when im keeping both `openlineage-airflow` and `apache-airflow-providers-openlineage` in my requirement file, i see the following error -\n``` from openlineage.airflow.extractors import Extractors\nModuleNotFoundError: No module named 'openlineage.airflow'```\nany thoughts?", + "user": "U062WLFMRTP", + "ts": "1698778838.540239", + "thread_ts": "1698340358.557159", + "root": { + "type": "message", + "text": "Im upgrading the version from openlineage-airflow==0.24.0 to openlineage-airflow 1.4.1 but im seeing the following error, any help is appreciated", + "files": [ + { + "id": "F062ZFJN2UB", + "created": 1698340299, + "timestamp": 1698340299, + "name": "Screenshot 2023-10-26 at 10.11.34 AM.png", + "title": "Screenshot 2023-10-26 at 10.11.34 AM.png", + "mimetype": "image/png", + "filetype": "png", + "pretty_type": "PNG", + "user": "U062WLFMRTP", + "user_team": "T01CWUYP5AR", + "editable": false, + "size": 356434, + "mode": "hosted", + "is_external": false, + "external_type": "", + "is_public": true, + "public_url_shared": false, + "display_as_bot": false, + "username": "", + "url_private": "https://files.slack.com/files-pri/T01CWUYP5AR-F062ZFJN2UB/screenshot_2023-10-26_at_10.11.34_am.png", + "url_private_download": "https://files.slack.com/files-pri/T01CWUYP5AR-F062ZFJN2UB/download/screenshot_2023-10-26_at_10.11.34_am.png", + "media_display_type": "unknown", + "thumb_64": "https://files.slack.com/files-tmb/T01CWUYP5AR-F062ZFJN2UB-0fca0a48ea/screenshot_2023-10-26_at_10.11.34_am_64.png", + "thumb_80": "https://files.slack.com/files-tmb/T01CWUYP5AR-F062ZFJN2UB-0fca0a48ea/screenshot_2023-10-26_at_10.11.34_am_80.png", + "thumb_360": "https://files.slack.com/files-tmb/T01CWUYP5AR-F062ZFJN2UB-0fca0a48ea/screenshot_2023-10-26_at_10.11.34_am_360.png", + "thumb_360_w": 360, + "thumb_360_h": 222, + "thumb_480": "https://files.slack.com/files-tmb/T01CWUYP5AR-F062ZFJN2UB-0fca0a48ea/screenshot_2023-10-26_at_10.11.34_am_480.png", + "thumb_480_w": 480, + "thumb_480_h": 297, + "thumb_160": "https://files.slack.com/files-tmb/T01CWUYP5AR-F062ZFJN2UB-0fca0a48ea/screenshot_2023-10-26_at_10.11.34_am_160.png", + "thumb_720": "https://files.slack.com/files-tmb/T01CWUYP5AR-F062ZFJN2UB-0fca0a48ea/screenshot_2023-10-26_at_10.11.34_am_720.png", + "thumb_720_w": 720, + "thumb_720_h": 445, + "thumb_800": "https://files.slack.com/files-tmb/T01CWUYP5AR-F062ZFJN2UB-0fca0a48ea/screenshot_2023-10-26_at_10.11.34_am_800.png", + "thumb_800_w": 800, + "thumb_800_h": 494, + "thumb_960": "https://files.slack.com/files-tmb/T01CWUYP5AR-F062ZFJN2UB-0fca0a48ea/screenshot_2023-10-26_at_10.11.34_am_960.png", + "thumb_960_w": 960, + "thumb_960_h": 593, + "thumb_1024": "https://files.slack.com/files-tmb/T01CWUYP5AR-F062ZFJN2UB-0fca0a48ea/screenshot_2023-10-26_at_10.11.34_am_1024.png", + "thumb_1024_w": 1024, + "thumb_1024_h": 633, + "original_w": 1756, + "original_h": 1085, + "thumb_tiny": "AwAdADC8etKKQ9acKADFLRRQAUUUUAMPWnCm96UUAOopM+1GaAFopM0ZoA//2Q==", + "permalink": "https://openlineage.slack.com/files/U062WLFMRTP/F062ZFJN2UB/screenshot_2023-10-26_at_10.11.34_am.png", + "permalink_public": "https://slack-files.com/T01CWUYP5AR-F062ZFJN2UB-c8de1a91b2", + "is_starred": false, + "has_rich_preview": false, + "file_access": "visible" + } + ], + "upload": false, + "user": "U062WLFMRTP", + "display_as_bot": false, + "ts": "1698340358.557159", + "blocks": [ + { + "type": "rich_text", + "block_id": "CRXyh", + "elements": [ + { + "type": "rich_text_section", + "elements": [ + { + "type": "text", + "text": "Im upgrading the version from openlineage-airflow==0.24.0 to openlineage-airflow 1.4.1 but im seeing the following error, any help is appreciated" + } + ] + } + ] + } + ], + "client_msg_id": "925bbcea-663c-4480-8809-2e2b3dd06020", + "thread_ts": "1698340358.557159", + "reply_count": 39, + "reply_users_count": 4, + "latest_reply": "1698786461.272129", + "reply_users": [ + "U062WLFMRTP", + "U02S6F54MAB", + "U01RA9B5GG2", + "U04AZ7992SU" + ], + "is_locked": false, + "subscribed": false + }, + "blocks": [ + { + "type": "rich_text", + "block_id": "7zvcN", + "elements": [ + { + "type": "rich_text_section", + "elements": [ + { + "type": "text", + "text": "one question if someone is around - when im keeping both " + }, + { + "type": "text", + "text": "openlineage-airflow", + "style": { + "code": true + } + }, + { + "type": "text", + "text": " and " + }, + { + "type": "text", + "text": "apache-airflow-providers-openlineage", + "style": { + "code": true + } + }, + { + "type": "text", + "text": " in my requirement file, i see the following error -\n" + } + ] + }, + { + "type": "rich_text_preformatted", + "elements": [ + { + "type": "text", + "text": " from openlineage.airflow.extractors import Extractors\nModuleNotFoundError: No module named 'openlineage.airflow'" + } + ], + "border": 0 + }, + { + "type": "rich_text_section", + "elements": [ + { + "type": "text", + "text": "any thoughts?" + } + ] + } + ] + } + ], + "client_msg_id": "84068da9-05c1-43a6-97a3-cb24d71c5832" + }, + { + "client_msg_id": "52f51233-521c-4a5d-a0c1-e6677707987c", + "type": "message", + "text": ":wave: Hi team, cross-posting from the Marquez Channel in case anyone here has a better idea of the spec\n\n> For most of our lineage extractors in airflow, we are using the rust sql parser from openlineage-sql to extract table lineage via sql statements. When errors occur we are adding an `extractionError` run facet similar to what is being done . I’m finding in the case that multiple statements were extracted but one failed to parse while many others were successful, the lineage for these runs doesn’t appear as expected in Marquez. Is there any logic around the `extractionError` run facet that could be causing this? It seems reasonable to assume that we might take this to mean the entire run event is invalid if we have any extraction errors. \n> \n> I would still expect to see the other lineage we sent for the run but am instead just seeing the `extractionError` in the marquez UI, in the database, runs with an `extractionError` facet don’t seem to make it to the `job_versions_io_mapping` table", + "user": "U04AZ7992SU", + "ts": "1698706303.956579", + "blocks": [ + { + "type": "rich_text", + "block_id": "532QI", + "elements": [ + { + "type": "rich_text_section", + "elements": [ + { + "type": "emoji", + "name": "wave", + "unicode": "1f44b" + }, + { + "type": "text", + "text": " Hi team, cross-posting from the Marquez Channel in case anyone here has a better idea of the spec\n\n" + } + ] + }, + { + "type": "rich_text_quote", + "elements": [ + { + "type": "text", + "text": "For most of our lineage extractors in airflow, we are using the rust sql parser from openlineage-sql to extract table lineage via sql statements. When errors occur we are adding an " + }, + { + "type": "text", + "text": "extractionError", + "style": { + "code": true + } + }, + { + "type": "text", + "text": " run facet similar to what is being done " + }, + { + "type": "link", + "url": "https://github.com/OpenLineage/OpenLineage/blob/main/integration/airflow/openlineage/airflow/extractors/sql_extractor.py#L75-L89", + "text": "here" + }, + { + "type": "text", + "text": ". I’m finding in the case that multiple statements were extracted but one failed to parse while many others were successful, the lineage for these runs doesn’t appear as expected in Marquez. Is there any logic around the " + }, + { + "type": "text", + "text": "extractionError", + "style": { + "code": true + } + }, + { + "type": "text", + "text": " run facet that could be causing this? It seems reasonable to assume that we might take this to mean the entire run event is invalid if we have any extraction errors. \n\nI would still expect to see the other lineage we sent for the run but am instead just seeing the " + }, + { + "type": "text", + "text": "extractionError", + "style": { + "code": true + } + }, + { + "type": "text", + "text": " in the marquez UI, in the database, runs with an " + }, + { + "type": "text", + "text": "extractionError", + "style": { + "code": true + } + }, + { + "type": "text", + "text": " facet don’t seem to make it to the " + }, + { + "type": "text", + "text": "job_versions_io_mapping", + "style": { + "code": true + } + }, + { + "type": "text", + "text": " table" + } + ] + } + ] + } + ], + "team": "T01CWUYP5AR", + "attachments": [ + { + "id": 1, + "footer_icon": "https://slack.github.com/static/img/favicon-neutral.png", + "color": "24292f", + "bot_id": "B01VA0FB340", + "app_unfurl_url": "https://github.com/OpenLineage/OpenLineage/blob/main/integration/airflow/openlineage/airflow/extractors/sql_extractor.py#L75-L89", + "is_app_unfurl": true, + "app_id": "A01BP7R4KNY", + "fallback": "", + "text": "```\n if sql_meta.errors:\n run_facets['extractionError'] = ExtractionErrorRunFacet(\n totalTasks=len(self.operator.sql) if isinstance(self.operator.sql, list) else 1,\n failedTasks=len(sql_meta.errors),\n errors=[\n ExtractionError(\n errorMessage=error.message,\n stackTrace=None,\n task=error.origin_statement,\n taskNumber=error.index\n )\n for error\n in sql_meta.errors\n ]\n )\n```", + "title": "", + "footer": "", + "mrkdwn_in": [ + "text" + ] + } + ], + "thread_ts": "1698706303.956579", + "reply_count": 11, + "reply_users_count": 3, + "latest_reply": "1698881826.588819", + "reply_users": [ + "U01RA9B5GG2", + "U05CAULTYG2", + "U04AZ7992SU" + ], + "is_locked": false, + "subscribed": false, + "replies": [ + { + "client_msg_id": "f364fc7b-1963-4ec3-8356-bd3952479e4f", + "type": "message", + "text": "Can you show the actual event? Should be in the events tab in Marquez", + "user": "U01RA9B5GG2", + "ts": "1698748445.166899", + "blocks": [ + { + "type": "rich_text", + "block_id": "6GDYD", + "elements": [ + { + "type": "rich_text_section", + "elements": [ + { + "type": "text", + "text": "Can you show the actual event? Should be in the events tab in Marquez" + } + ] + } + ] + } + ], + "team": "T01CWUYP5AR", + "thread_ts": "1698706303.956579", + "parent_user_id": "U04AZ7992SU" + }, + { + "client_msg_id": "78ff6b9b-79ea-4eb0-9878-e2d3a468bdb2", + "type": "message", + "text": "<@U04AZ7992SU>, would you mind posting the link to Marquez teams slack channel?", + "user": "U05CAULTYG2", + "ts": "1698767947.720639", + "blocks": [ + { + "type": "rich_text", + "block_id": "uFKlN", + "elements": [ + { + "type": "rich_text_section", + "elements": [ + { + "type": "user", + "user_id": "U04AZ7992SU" + }, + { + "type": "text", + "text": ", would you mind posting the link to Marquez teams slack channel?" + } + ] + } + ] + } + ], + "team": "T01CWUYP5AR", + "thread_ts": "1698706303.956579", + "parent_user_id": "U04AZ7992SU" + }, + { + "client_msg_id": "224705df-657a-4c1f-a73a-cdfe83bda101", + "type": "message", + "text": "yep here is the link: \n\nThis is the full event, sanitized of internal info:\n```{\n \"job\": {\n \"name\": \"some_dag.some_task\",\n \"facets\": {},\n \"namespace\": \"default\"\n },\n \"run\": {\n \"runId\": \"a9565df2-f1a1-3ee3-b202-7626f8c4b92d\",\n \"facets\": {\n \"extractionError\": {\n \"errors\": [\n {\n \"task\": \"ALTER SESSION UNSET QUERY_TAG;\",\n \"_producer\": \"\",\n \"_schemaURL\": \"\",\n \"taskNumber\": 0,\n \"errorMessage\": \"Expected one of TABLE or INDEX, found: SESSION\"\n }\n ],\n \"_producer\": \"\",\n \"_schemaURL\": \"\",\n \"totalTasks\": 1,\n \"failedTasks\": 1\n }\n }\n },\n \"inputs\": [\n {\n \"name\": \"foo.bar\",\n \"facets\": {},\n \"namespace\": \"snowflake\"\n },\n {\n \"name\": \"fizz.buzz\",\n \"facets\": {},\n \"namespace\": \"snowflake\"\n }\n ],\n \"outputs\": [\n { \"name\": \"foo1.bar2\", \"facets\": {}, \"namespace\": \"snowflake\" },\n {\n \"name\": \"fizz1.buzz2\",\n \"facets\": {},\n \"namespace\": \"snowflake\"\n }\n ],\n \"producer\": \"\",\n \"eventTime\": \"2023-10-30T02:46:13.367274Z\",\n \"eventType\": \"COMPLETE\"\n}```", + "user": "U04AZ7992SU", + "ts": "1698768937.061429", + "blocks": [ + { + "type": "rich_text", + "block_id": "xChGR", + "elements": [ + { + "type": "rich_text_section", + "elements": [ + { + "type": "text", + "text": "yep here is the link: " + }, + { + "type": "link", + "url": "https://marquezproject.slack.com/archives/C01E8MQGJP7/p1698702140709439" + }, + { + "type": "text", + "text": "\n\nThis is the full event, sanitized of internal info:\n" + } + ] + }, + { + "type": "rich_text_preformatted", + "elements": [ + { + "type": "text", + "text": "{\n \"job\": {\n \"name\": \"some_dag.some_task\",\n \"facets\": {},\n \"namespace\": \"default\"\n },\n \"run\": {\n \"runId\": \"a9565df2-f1a1-3ee3-b202-7626f8c4b92d\",\n \"facets\": {\n \"extractionError\": {\n \"errors\": [\n {\n \"task\": \"ALTER SESSION UNSET QUERY_TAG;\",\n \"_producer\": \"" + }, + { + "type": "link", + "url": "https://github.com/OpenLineage/OpenLineage/tree/0.24.0/client/python" + }, + { + "type": "text", + "text": "\",\n \"_schemaURL\": \"" + }, + { + "type": "link", + "url": "https://raw.githubusercontent.com/OpenLineage/OpenLineage/main/spec/OpenLineage.json#/definitions/BaseFacet" + }, + { + "type": "text", + "text": "\",\n \"taskNumber\": 0,\n \"errorMessage\": \"Expected one of TABLE or INDEX, found: SESSION\"\n }\n ],\n \"_producer\": \"" + }, + { + "type": "link", + "url": "https://github.com/OpenLineage/OpenLineage/tree/0.24.0/client/python" + }, + { + "type": "text", + "text": "\",\n \"_schemaURL\": \"" + }, + { + "type": "link", + "url": "https://raw.githubusercontent.com/OpenLineage/OpenLineage/main/spec/OpenLineage.json#/definitions/ExtractionErrorRunFacet" + }, + { + "type": "text", + "text": "\",\n \"totalTasks\": 1,\n \"failedTasks\": 1\n }\n }\n },\n \"inputs\": [\n {\n \"name\": \"foo.bar\",\n \"facets\": {},\n \"namespace\": \"snowflake\"\n },\n {\n \"name\": \"fizz.buzz\",\n \"facets\": {},\n \"namespace\": \"snowflake\"\n }\n ],\n \"outputs\": [\n { \"name\": \"foo1.bar2\", \"facets\": {}, \"namespace\": \"snowflake\" },\n {\n \"name\": \"fizz1.buzz2\",\n \"facets\": {},\n \"namespace\": \"snowflake\"\n }\n ],\n \"producer\": \"" + }, + { + "type": "link", + "url": "https://github.com/MyCompany/repo/blob/next-master/company/data/pipelines/airflow_utils/openlineage_utils/client.py" + }, + { + "type": "text", + "text": "\",\n \"eventTime\": \"2023-10-30T02:46:13.367274Z\",\n \"eventType\": \"COMPLETE\"\n}" + } + ], + "border": 0 + } + ] + } + ], + "team": "T01CWUYP5AR", + "thread_ts": "1698706303.956579", + "parent_user_id": "U04AZ7992SU" + }, + { + "client_msg_id": "dd5e58c1-61aa-43ce-b90b-3677f05f8a77", + "type": "message", + "text": "thank you!", + "user": "U05CAULTYG2", + "ts": "1698770587.611649", + "blocks": [ + { + "type": "rich_text", + "block_id": "afEI5", + "elements": [ + { + "type": "rich_text_section", + "elements": [ + { + "type": "text", + "text": "thank you!" + } + ] + } + ] + } + ], + "team": "T01CWUYP5AR", + "thread_ts": "1698706303.956579", + "parent_user_id": "U04AZ7992SU" + }, + { + "client_msg_id": "dc2ce936-1acb-4730-aeb9-bcc5bb3ae25a", + "type": "message", + "text": "<@U04AZ7992SU>, sorry to trouble again, is the slack channel still active? for whatever reason i cant get to this workspace", + "user": "U05CAULTYG2", + "ts": "1698772469.402859", + "blocks": [ + { + "type": "rich_text", + "block_id": "77tAM", + "elements": [ + { + "type": "rich_text_section", + "elements": [ + { + "type": "user", + "user_id": "U04AZ7992SU" + }, + { + "type": "text", + "text": ", sorry to trouble again, is the slack channel still active? for whatever reason i cant get to this workspace" + } + ] + } + ] + } + ], + "team": "T01CWUYP5AR", + "edited": { + "user": "U05CAULTYG2", + "ts": "1698777868.000000" + }, + "thread_ts": "1698706303.956579", + "parent_user_id": "U04AZ7992SU" + }, + { + "client_msg_id": "d5e0beb7-981d-470f-8b61-8c64f2ca47dc", + "type": "message", + "text": "yep it’s still active, maybe you need to join the workspace first? ", + "user": "U04AZ7992SU", + "ts": "1698772526.341899", + "blocks": [ + { + "type": "rich_text", + "block_id": "Ckj1E", + "elements": [ + { + "type": "rich_text_section", + "elements": [ + { + "type": "text", + "text": "yep it’s still active, maybe you need to join the workspace first? " + }, + { + "type": "link", + "url": "https://join.slack.com/t/marquezproject/shared_invite/zt-266fdhg9g-TE7e0p~EHK50GJMMqNH4tg" + } + ] + } + ] + } + ], + "team": "T01CWUYP5AR", + "thread_ts": "1698706303.956579", + "parent_user_id": "U04AZ7992SU" + }, + { + "client_msg_id": "f612b95d-4ddb-4a3d-9a37-b69951d280e6", + "type": "message", + "text": "that was a good call. the link you just shared worked! thank you!", + "user": "U05CAULTYG2", + "ts": "1698773151.879409", + "blocks": [ + { + "type": "rich_text", + "block_id": "mD/jk", + "elements": [ + { + "type": "rich_text_section", + "elements": [ + { + "type": "text", + "text": "that was a good call. the link you just shared worked! thank you!" + } + ] + } + ] + } + ], + "team": "T01CWUYP5AR", + "thread_ts": "1698706303.956579", + "parent_user_id": "U04AZ7992SU" + }, + { + "client_msg_id": "2607d391-2a1a-4769-b713-70637f34577c", + "type": "message", + "text": "yeah from OL perspective this looks good - the inputs and outputs are there, the extraction error facet looks like it should", + "user": "U01RA9B5GG2", + "ts": "1698773275.821549", + "blocks": [ + { + "type": "rich_text", + "block_id": "Akb2+", + "elements": [ + { + "type": "rich_text_section", + "elements": [ + { + "type": "text", + "text": "yeah from OL perspective this looks good - the inputs and outputs are there, the extraction error facet looks like it should" + } + ] + } + ] + } + ], + "team": "T01CWUYP5AR", + "thread_ts": "1698706303.956579", + "parent_user_id": "U04AZ7992SU" + }, + { + "client_msg_id": "147fd829-ef27-4f73-8771-44e60e41ba70", + "type": "message", + "text": "must be some Marquez hiccup :slightly_smiling_face:", + "user": "U01RA9B5GG2", + "ts": "1698773285.952189", + "blocks": [ + { + "type": "rich_text", + "block_id": "qvUWQ", + "elements": [ + { + "type": "rich_text_section", + "elements": [ + { + "type": "text", + "text": "must be some Marquez hiccup " + }, + { + "type": "emoji", + "name": "slightly_smiling_face", + "unicode": "1f642" + } + ] + } + ] + } + ], + "team": "T01CWUYP5AR", + "thread_ts": "1698706303.956579", + "parent_user_id": "U04AZ7992SU", + "reactions": [ + { + "name": "+1", + "users": [ + "U04AZ7992SU" + ], + "count": 1 + } + ] + }, + { + "client_msg_id": "a67feb72-4b89-48b2-9e82-067682c3cca1", + "type": "message", + "text": "Makes sense, I’ll tail my marquez logs today to see if I can find anything", + "user": "U04AZ7992SU", + "ts": "1698773325.668029", + "blocks": [ + { + "type": "rich_text", + "block_id": "7PgFA", + "elements": [ + { + "type": "rich_text_section", + "elements": [ + { + "type": "text", + "text": "Makes sense, I’ll tail my marquez logs today to see if I can find anything" + } + ] + } + ] + } + ], + "team": "T01CWUYP5AR", + "thread_ts": "1698706303.956579", + "parent_user_id": "U04AZ7992SU" + }, + { + "client_msg_id": "423e464e-4fab-4f39-ad13-22b8ab8068c0", + "type": "message", + "text": "Somehow this started working after we switched from our beta to prod infrastructure. I suspect something was failing due to constraints on the size of our db and the load of poor quality data it was under after months of testing against it", + "user": "U04AZ7992SU", + "ts": "1698881826.588819", + "blocks": [ + { + "type": "rich_text", + "block_id": "wvBxx", + "elements": [ + { + "type": "rich_text_section", + "elements": [ + { + "type": "text", + "text": "Somehow this started working after we switched from our beta to prod infrastructure. I suspect something was failing due to constraints on the size of our db and the load of poor quality data it was under after months of testing against it" + } + ] + } + ] + } + ], + "team": "T01CWUYP5AR", + "thread_ts": "1698706303.956579", + "parent_user_id": "U04AZ7992SU" + } + ] + }, + { + "client_msg_id": "80b66ffa-7be5-4f30-9bf2-d5c6af7abfde", + "type": "message", + "text": "I realize in Spark 3.4+, some job ids don't have a start event. What part of the code is responsible for triggering the START and COMPLETE event", + "user": "U05T8BJD4DU", + "ts": "1698563188.319939", + "blocks": [ + { + "type": "rich_text", + "block_id": "LpAAj", + "elements": [ + { + "type": "rich_text_section", + "elements": [ + { + "type": "text", + "text": "I realize in Spark 3.4+, some job ids don't have a start event. What part of the code is responsible for triggering the START and COMPLETE event" + } + ] + } + ] + } + ], + "team": "T01CWUYP5AR", + "edited": { + "user": "U05T8BJD4DU", + "ts": "1698563291.000000" + }, + "thread_ts": "1698563188.319939", + "reply_count": 2, + "reply_users_count": 2, + "latest_reply": "1698699115.599449", + "reply_users": [ + "U02MK6YNAQ5", + "U05T8BJD4DU" + ], + "is_locked": false, + "subscribed": false, + "replies": [ + { + "client_msg_id": "d78c6581-7c33-43e0-adc2-05ff7e3d131d", + "type": "message", + "text": "hi <@U05T8BJD4DU> could you provide an example of such a job?", + "user": "U02MK6YNAQ5", + "ts": "1698674393.317219", + "blocks": [ + { + "type": "rich_text", + "block_id": "bd1pS", + "elements": [ + { + "type": "rich_text_section", + "elements": [ + { + "type": "text", + "text": "hi " + }, + { + "type": "user", + "user_id": "U05T8BJD4DU" + }, + { + "type": "text", + "text": " could you provide an example of such a job?" + } + ] + } + ] + } + ], + "team": "T01CWUYP5AR", + "thread_ts": "1698563188.319939", + "parent_user_id": "U05T8BJD4DU" + }, + { + "client_msg_id": "1d4ef7b2-bcdc-49f4-ac5a-df4a25d4251a", + "type": "message", + "text": "<@U02MK6YNAQ5> same old:\n\n# delete the old table if needed\n_ = spark.sql('DROP TABLE IF EXISTS transactions')\n\n# expected structure of the file\ntransactions_schema = StructType([\n StructField('household_id', IntegerType()),\n StructField('basket_id', LongType()),\n StructField('day', IntegerType()),\n StructField('product_id', IntegerType()),\n StructField('quantity', IntegerType()),\n StructField('sales_amount', FloatType()),\n StructField('store_id', IntegerType()),\n StructField('discount_amount', FloatType()),\n StructField('transaction_time', IntegerType()),\n StructField('week_no', IntegerType()),\n StructField('coupon_discount', FloatType()),\n StructField('coupon_discount_match', FloatType())\n ])\n\n# read data to dataframe\ndf = (spark\n .read\n .csv(\n adlsRootPath + '/examples/data/csv/completejourney/transaction_data.csv',\n header=True,\n schema=transactions_schema))\n\ndf.write\\\n .format('delta')\\\n .mode('overwrite')\\\n .option('overwriteSchema', 'true')\\\n .option('path', adlsRootPath + '/examples/data/csv/completejourney/silver/transactions')\\\n .saveAsTable('transactions')\n\ndf.count()\n\n# # create table object to make delta lake queryable\n# _ = spark.sql(f'''\n# CREATE TABLE transactions\n# USING DELTA\n# LOCATION '{adlsRootPath}/examples/data/csv/completejourney/silver/transactions'\n# ''')\n\n# show data\ndisplay(\n spark.table('transactions')\n )", + "user": "U05T8BJD4DU", + "ts": "1698699115.599449", + "blocks": [ + { + "type": "rich_text", + "block_id": "2gImX", + "elements": [ + { + "type": "rich_text_section", + "elements": [ + { + "type": "user", + "user_id": "U02MK6YNAQ5" + }, + { + "type": "text", + "text": " same old:\n\n# delete the old table if needed\n_ = spark.sql('DROP TABLE IF EXISTS transactions')\n\n# expected structure of the file\ntransactions_schema = StructType([\n StructField('household_id', IntegerType()),\n StructField('basket_id', LongType()),\n StructField('day', IntegerType()),\n StructField('product_id', IntegerType()),\n StructField('quantity', IntegerType()),\n StructField('sales_amount', FloatType()),\n StructField('store_id', IntegerType()),\n StructField('discount_amount', FloatType()),\n StructField('transaction_time', IntegerType()),\n StructField('week_no', IntegerType()),\n StructField('coupon_discount', FloatType()),\n StructField('coupon_discount_match', FloatType())\n ])\n\n# read data to dataframe\ndf = (spark\n .read\n .csv(\n adlsRootPath + '/examples/data/csv/completejourney/transaction_data.csv',\n header=True,\n schema=transactions_schema))\n\ndf.write\\\n .format('delta')\\\n .mode('overwrite')\\\n .option('overwriteSchema', 'true')\\\n .option('path', adlsRootPath + '/examples/data/csv/completejourney/silver/transactions')\\\n .saveAsTable('transactions')\n\ndf.count()\n\n# # create table object to make delta lake queryable\n# _ = spark.sql(f'''\n# CREATE TABLE transactions\n# USING DELTA\n# LOCATION '{adlsRootPath}/examples/data/csv/completejourney/silver/transactions'\n# ''')\n\n# show data\ndisplay(\n spark.table('transactions')\n )" + } + ] + } + ] + } + ], + "team": "T01CWUYP5AR", + "thread_ts": "1698563188.319939", + "parent_user_id": "U05T8BJD4DU" + } + ] + }, + { + "client_msg_id": "f5fddb24-8cfd-4510-8b01-8c70061bb054", + "type": "message", + "text": "Hello, has anyone run into similar error as posted in this github open issues[] while setting up marquez on an EC2 Instance, would appreciate any help to get past the errors", + "user": "U05CAULTYG2", + "ts": "1698440472.145489", + "blocks": [ + { + "type": "rich_text", + "block_id": "Ftpj6", + "elements": [ + { + "type": "rich_text_section", + "elements": [ + { + "type": "text", + "text": "Hello, has anyone run into similar error as posted in this github open issues[" + }, + { + "type": "link", + "url": "https://github.com/MarquezProject/marquez/issues/2468" + }, + { + "type": "text", + "text": "] while setting up marquez on an EC2 Instance, would appreciate any help to get past the errors" + } + ] + } + ] + } + ], + "team": "T01CWUYP5AR", + "attachments": [ + { + "id": 1, + "footer_icon": "https://slack.github.com/static/img/favicon-neutral.png", + "ts": 1680563711, + "color": "36a64f", + "bot_id": "B01VA0FB340", + "app_unfurl_url": "https://github.com/MarquezProject/marquez/issues/2468", + "is_app_unfurl": true, + "app_id": "A01BP7R4KNY", + "fallback": "#2468 Connection error PostgreSQL and Marquez", + "text": "Hi all!\n\nI really appreciate this repository and would love to use it for its data lineage aspect! \nLast few days, I've been trying to get the `marquez api` Docker image up and running but whatever I try, it does not seem to work.\n\nIt returns the following errors in Docker Desktop:\n\n`2023-04-03 18:07:04 marquez-db | 2023-04-03 23:07:04.726 GMT [35] FATAL: password authentication failed for user \"marquez\" 2023-04-03 18:07:04 marquez-db | 2023-04-03 23:07:04.726 GMT [35] DETAIL: Role \"marquez\" does not exist.`\n\nand\n\n`ERROR [2023-04-03 22:44:31,217] org.apache.tomcat.jdbc.pool.ConnectionPool: Unable to create initial connections of pool. 2023-04-03 17:44:31 ! org.postgresql.util.PSQLException: FATAL: password authentication failed for user \"marquez\"`\n\nAll the steps in the README.md file were used to configure the PostgreSQL database for Marquez and to set the correct environment variables. \nDuring my trouble shooting practices, I found a relatively similar issue on StackOverflow which refers to the danger in using the same ports for Postgres and Marquez but this did not help me yet in solving the issue ().\n\nCould you please help me out?\n\nKind regards,\n\nTom", + "title": "#2468 Connection error PostgreSQL and Marquez", + "title_link": "https://github.com/MarquezProject/marquez/issues/2468", + "footer": "", + "fields": [ + { + "value": "5", + "title": "Comments", + "short": true + } + ], + "mrkdwn_in": [ + "text" + ] + } + ], + "thread_ts": "1698440472.145489", + "reply_count": 33, + "reply_users_count": 2, + "latest_reply": "1698453369.842939", + "reply_users": [ + "U01DCMDFHBK", + "U05CAULTYG2" + ], + "is_locked": false, + "subscribed": false, + "replies": [ + { + "client_msg_id": "2d34aa83-0a6c-4509-9577-cb6685597919", + "type": "message", + "text": "Hmm, have you looked over our docs?", + "user": "U01DCMDFHBK", + "ts": "1698440670.762299", + "blocks": [ + { + "type": "rich_text", + "block_id": "tYvDg", + "elements": [ + { + "type": "rich_text_section", + "elements": [ + { + "type": "text", + "text": "Hmm, have you looked over our " + }, + { + "type": "link", + "url": "https://marquezproject.ai/docs/deployment/running-on-aws", + "text": "Running on AWS" + }, + { + "type": "text", + "text": " docs?" + } + ] + } + ] + } + ], + "team": "T01CWUYP5AR", + "thread_ts": "1698440472.145489", + "parent_user_id": "U05CAULTYG2" + }, + { + "client_msg_id": "cb6742ec-9a22-485f-8542-b87b1d3cc465", + "type": "message", + "text": "More specifically, the AWS RDS section. How are you deploying Marquez on Ec2?", + "user": "U01DCMDFHBK", + "ts": "1698440768.525519", + "blocks": [ + { + "type": "rich_text", + "block_id": "4lIXN", + "elements": [ + { + "type": "rich_text_section", + "elements": [ + { + "type": "text", + "text": "More specifically, the AWS RDS section. How are you deploying Marquez on Ec2?" + } + ] + } + ] + } + ], + "team": "T01CWUYP5AR", + "thread_ts": "1698440472.145489", + "parent_user_id": "U05CAULTYG2" + }, + { + "client_msg_id": "ee6b2c14-106d-4e0c-8978-d098ca03d902", + "type": "message", + "text": "we were primarily referencing this document on git - ", + "user": "U05CAULTYG2", + "ts": "1698440885.315539", + "blocks": [ + { + "type": "rich_text", + "block_id": "jUo+G", + "elements": [ + { + "type": "rich_text_section", + "elements": [ + { + "type": "text", + "text": "we were primarily referencing this document on git - " + }, + { + "type": "link", + "url": "https://github.com/MarquezProject/marquez" + } + ] + } + ] + } + ], + "team": "T01CWUYP5AR", + "attachments": [ + { + "id": 1, + "color": "24292f", + "bot_id": "B01VA0FB340", + "app_unfurl_url": "https://github.com/MarquezProject/marquez", + "is_app_unfurl": true, + "app_id": "A01BP7R4KNY", + "fallback": "MarquezProject/marquez", + "text": "Collect, aggregate, and visualize a data ecosystem's metadata", + "title": "MarquezProject/marquez", + "fields": [ + { + "value": "", + "title": "Website", + "short": true + }, + { + "value": "1450", + "title": "Stars", + "short": true + } + ] + } + ], + "thread_ts": "1698440472.145489", + "parent_user_id": "U05CAULTYG2" + }, + { + "client_msg_id": "f564f3a6-19b6-4e3a-86f1-b289b385bf20", + "type": "message", + "text": "leveraged docker and docker-compose", + "user": "U05CAULTYG2", + "ts": "1698440945.117219", + "blocks": [ + { + "type": "rich_text", + "block_id": "jn0Hq", + "elements": [ + { + "type": "rich_text_section", + "elements": [ + { + "type": "text", + "text": "leveraged docker and docker-compose" + } + ] + } + ] + } + ], + "team": "T01CWUYP5AR", + "thread_ts": "1698440472.145489", + "parent_user_id": "U05CAULTYG2" + }, + { + "client_msg_id": "640a7548-04df-4d37-9e34-ad60b8e4dbea", + "type": "message", + "text": "hmm so you’re running docker-compose up on an Ec2 instance you’ve ssh’d into? (just trying to understand your setup better)", + "user": "U01DCMDFHBK", + "ts": "1698441190.356069", + "blocks": [ + { + "type": "rich_text", + "block_id": "XDyez", + "elements": [ + { + "type": "rich_text_section", + "elements": [ + { + "type": "text", + "text": "hmm so you’re running docker-compose up on an Ec2 instance you’ve ssh’d into? (just trying to understand your setup better)" + } + ] + } + ] + } + ], + "team": "T01CWUYP5AR", + "thread_ts": "1698440472.145489", + "parent_user_id": "U05CAULTYG2" + }, + { + "client_msg_id": "1872d1ac-0d97-4627-be45-3f719c4bd8cb", + "type": "message", + "text": "yes, thats correct", + "user": "U05CAULTYG2", + "ts": "1698441206.923589", + "blocks": [ + { + "type": "rich_text", + "block_id": "uuZAU", + "elements": [ + { + "type": "rich_text_section", + "elements": [ + { + "type": "text", + "text": "yes, thats correct" + } + ] + } + ] + } + ], + "team": "T01CWUYP5AR", + "thread_ts": "1698440472.145489", + "parent_user_id": "U05CAULTYG2" + }, + { + "client_msg_id": "8003944a-a989-4a14-9c78-150d02232b97", + "type": "message", + "text": "I’ve only used docker compose for local dev or integration tests. but, ok you’re probably in the PoC phase. Can you run the docker cmd on you local machine successfully? What OS is stalled on the Ec2 instance?", + "user": "U01DCMDFHBK", + "ts": "1698441399.016209", + "blocks": [ + { + "type": "rich_text", + "block_id": "yy+k/", + "elements": [ + { + "type": "rich_text_section", + "elements": [ + { + "type": "text", + "text": "I’ve only used docker compose for local dev or integration tests. but, ok you’re probably in the PoC phase. Can you run the docker cmd on you local machine successfully? What OS is stalled on the Ec2 instance?" + } + ] + } + ] + } + ], + "team": "T01CWUYP5AR", + "edited": { + "user": "U01DCMDFHBK", + "ts": "1698441405.000000" + }, + "thread_ts": "1698440472.145489", + "parent_user_id": "U05CAULTYG2" + }, + { + "client_msg_id": "abeb855c-681e-4f12-b3dc-b8037873bffc", + "type": "message", + "text": "yes, i can run and the OS is Ubuntu 20.04.6 LTS", + "user": "U05CAULTYG2", + "ts": "1698441480.358829", + "blocks": [ + { + "type": "rich_text", + "block_id": "VijLp", + "elements": [ + { + "type": "rich_text_section", + "elements": [ + { + "type": "text", + "text": "yes, i can run and the OS is Ubuntu 20.04.6 LTS" + } + ] + } + ] + } + ], + "team": "T01CWUYP5AR", + "thread_ts": "1698440472.145489", + "parent_user_id": "U05CAULTYG2" + }, + { + "client_msg_id": "24604801-85ba-45b1-a4dd-89ddbe04bc57", + "type": "message", + "text": "we initiallly ran into a permission denied error related to postgressql.conf file and we had to update file permissions to 777 and after which we started to see below errors", + "user": "U05CAULTYG2", + "ts": "1698441567.903199", + "blocks": [ + { + "type": "rich_text", + "block_id": "ZzdHw", + "elements": [ + { + "type": "rich_text_section", + "elements": [ + { + "type": "text", + "text": "we initiallly ran into a permission denied error related to postgressql.conf file and we had to update file permissions to 777 and after which we started to see below errors" + } + ] + } + ] + } + ], + "team": "T01CWUYP5AR", + "thread_ts": "1698440472.145489", + "parent_user_id": "U05CAULTYG2" + }, + { + "client_msg_id": "dcdd7b78-918e-49e6-9a9f-d36625ee3be7", + "type": "message", + "text": "marquez-db | 2023-10-27 20:35:52.512 GMT [35] FATAL: no pg_hba.conf entry for host \"172.18.0.5\", user \"marquez\", database \"marquez\", no encryption\n marquez-db | 2023-10-27 20:35:52.529 GMT [36] FATAL: no pg_hba.conf entry for host \"172.18.0.5\", user \"marquez\", database \"marquez\", no encryption", + "user": "U05CAULTYG2", + "ts": "1698441576.039939", + "blocks": [ + { + "type": "rich_text", + "block_id": "g0oZR", + "elements": [ + { + "type": "rich_text_section", + "elements": [ + { + "type": "text", + "text": "marquez-db | 2023-10-27 20:35:52.512 GMT [35] FATAL: no pg_hba.conf entry for host \"172.18.0.5\", user \"marquez\", database \"marquez\", no encryption\n marquez-db | 2023-10-27 20:35:52.529 GMT [36] FATAL: no pg_hba.conf entry for host \"172.18.0.5\", user \"marquez\", database \"marquez\", no encryption" + } + ] + } + ] + } + ], + "team": "T01CWUYP5AR", + "thread_ts": "1698440472.145489", + "parent_user_id": "U05CAULTYG2" + }, + { + "client_msg_id": "edfaac42-b2a9-4041-9d56-75e0714cbfbd", + "type": "message", + "text": "we then manually updated pg_hba.conf file to include host user and db details", + "user": "U05CAULTYG2", + "ts": "1698441612.909429", + "blocks": [ + { + "type": "rich_text", + "block_id": "x3bgz", + "elements": [ + { + "type": "rich_text_section", + "elements": [ + { + "type": "text", + "text": "we then manually updated pg_hba.conf file to include host user and db details" + } + ] + } + ] + } + ], + "team": "T01CWUYP5AR", + "thread_ts": "1698440472.145489", + "parent_user_id": "U05CAULTYG2" + }, + { + "client_msg_id": "d29f12c2-4054-4b75-9fc8-cdfd29b95bbd", + "type": "message", + "text": "Did you also update the `marquez.yml` with the db user / password?", + "user": "U01DCMDFHBK", + "ts": "1698441642.635089", + "blocks": [ + { + "type": "rich_text", + "block_id": "KynpH", + "elements": [ + { + "type": "rich_text_section", + "elements": [ + { + "type": "text", + "text": "Did you also update the " + }, + { + "type": "text", + "text": "marquez.yml", + "style": { + "code": true + } + }, + { + "type": "text", + "text": " with the db user / password?" + } + ] + } + ] + } + ], + "team": "T01CWUYP5AR", + "thread_ts": "1698440472.145489", + "parent_user_id": "U05CAULTYG2" + }, + { + "client_msg_id": "e28dc315-eddd-4ef6-ad18-1b8a3ac9a007", + "type": "message", + "text": "after which we started to see the errors posted in the github open issues page", + "user": "U05CAULTYG2", + "ts": "1698441648.000319", + "blocks": [ + { + "type": "rich_text", + "block_id": "B/I8C", + "elements": [ + { + "type": "rich_text_section", + "elements": [ + { + "type": "text", + "text": "after which we started to see the errors posted in the github open issues page" + } + ] + } + ] + } + ], + "team": "T01CWUYP5AR", + "thread_ts": "1698440472.145489", + "parent_user_id": "U05CAULTYG2" + }, + { + "client_msg_id": "6352952a-445b-449e-bf05-6e1ed6f45f59", + "type": "message", + "text": "hmm are you using an external database or are you spinning up the entire Marquez stack with docker compose?", + "user": "U01DCMDFHBK", + "ts": "1698441693.285399", + "blocks": [ + { + "type": "rich_text", + "block_id": "jgcBs", + "elements": [ + { + "type": "rich_text_section", + "elements": [ + { + "type": "text", + "text": "hmm are you using an external database or are you spinning up the entire Marquez stack with docker compose?" + } + ] + } + ] + } + ], + "team": "T01CWUYP5AR", + "thread_ts": "1698440472.145489", + "parent_user_id": "U05CAULTYG2" + }, + { + "client_msg_id": "a951c03a-b61d-4544-b053-da995814c476", + "type": "message", + "text": "we are spinning up the entire Marquez stack with docker compose", + "user": "U05CAULTYG2", + "ts": "1698441716.256449", + "blocks": [ + { + "type": "rich_text", + "block_id": "uzxNS", + "elements": [ + { + "type": "rich_text_section", + "elements": [ + { + "type": "text", + "text": "we are spinning up the entire Marquez stack with docker compose" + } + ] + } + ] + } + ], + "team": "T01CWUYP5AR", + "thread_ts": "1698440472.145489", + "parent_user_id": "U05CAULTYG2" + }, + { + "client_msg_id": "38ba244b-7896-416b-ae31-3d521c0c7295", + "type": "message", + "text": "we did not change anything in the marquez.yml, i think we did not find that file in the github repo that we cloned into our local instance", + "user": "U05CAULTYG2", + "ts": "1698441804.724249", + "blocks": [ + { + "type": "rich_text", + "block_id": "VCqMj", + "elements": [ + { + "type": "rich_text_section", + "elements": [ + { + "type": "text", + "text": "we did not change anything in the marquez.yml, i think we did not find that file in the github repo that we cloned into our local instance" + } + ] + } + ] + } + ], + "team": "T01CWUYP5AR", + "edited": { + "user": "U05CAULTYG2", + "ts": "1698441835.000000" + }, + "thread_ts": "1698440472.145489", + "parent_user_id": "U05CAULTYG2" + }, + { + "client_msg_id": "e698c011-e929-447f-a352-9ec8429f6969", + "type": "message", + "text": "It’s important that the script runs, but I don’t think it is", + "user": "U01DCMDFHBK", + "ts": "1698441991.492989", + "blocks": [ + { + "type": "rich_text", + "block_id": "GZZrk", + "elements": [ + { + "type": "rich_text_section", + "elements": [ + { + "type": "text", + "text": "It’s important that the " + }, + { + "type": "link", + "url": "https://github.com/MarquezProject/marquez/blob/main/docker/init-db.sh", + "text": "init-db.sh", + "unsafe": true + }, + { + "type": "text", + "text": " script runs, but I don’t think it is" + } + ] + } + ] + } + ], + "team": "T01CWUYP5AR", + "thread_ts": "1698440472.145489", + "parent_user_id": "U05CAULTYG2" + }, + { + "client_msg_id": "65773a50-eaff-475e-9ca7-2dde2df541c9", + "type": "message", + "text": "can you grab all the docker compose logs and share them? it’s hard to debug otherwise", + "user": "U01DCMDFHBK", + "ts": "1698442016.355249", + "blocks": [ + { + "type": "rich_text", + "block_id": "DKq6n", + "elements": [ + { + "type": "rich_text_section", + "elements": [ + { + "type": "text", + "text": "can you grab all the docker compose logs and share them? it’s hard to debug otherwise" + } + ] + } + ] + } + ], + "team": "T01CWUYP5AR", + "edited": { + "user": "U01DCMDFHBK", + "ts": "1698442042.000000" + }, + "thread_ts": "1698440472.145489", + "parent_user_id": "U05CAULTYG2" + }, + { + "type": "message", + "text": "", + "files": [ + { + "id": "F06375RT0LS", + "created": 1698442196, + "timestamp": 1698442196, + "name": "logs.txt", + "title": "logs.txt", + "mimetype": "text/plain", + "filetype": "text", + "pretty_type": "Plain Text", + "user": "U05CAULTYG2", + "user_team": "T01CWUYP5AR", + "editable": true, + "size": 24143, + "mode": "snippet", + "is_external": false, + "external_type": "", + "is_public": true, + "public_url_shared": false, + "display_as_bot": false, + "username": "", + "url_private": "https://files.slack.com/files-pri/T01CWUYP5AR-F06375RT0LS/logs.txt", + "url_private_download": "https://files.slack.com/files-pri/T01CWUYP5AR-F06375RT0LS/download/logs.txt", + "permalink": "https://openlineage.slack.com/files/U05CAULTYG2/F06375RT0LS/logs.txt", + "permalink_public": "https://slack-files.com/T01CWUYP5AR-F06375RT0LS-4134397fd8", + "edit_link": "https://openlineage.slack.com/files/U05CAULTYG2/F06375RT0LS/logs.txt/edit", + "preview": "root@ip-172-30-4-153:~/marquez# ./docker/up.sh --tag 0.37.0 -a 5000 -m 5001 -w 3000 --build\r\n...creating volumes: marquez_data, marquez_db-conf, marquez_db-init, marquez_db-backup\r\nSuccessfully copied 7.17kB to volumes-provisioner:/data/wait-for-it.sh\r\nAdded files to volume marquez_data: wait-for-it.sh\r\nSuccessfully copied 2.05kB to volumes-provisioner:/db-conf/postgresql.conf\r", + "preview_highlight": "
\n
\n
root@ip-172-30-4-153:~/marquez# ./docker/up.sh --tag 0.37.0 -a 5000 -m 5001 -w 3000 --build
\n
...creating volumes: marquez_data, marquez_db-conf, marquez_db-init, marquez_db-backup
\n
Successfully copied 7.17kB to volumes-provisioner:/data/wait-for-it.sh
\n
Added files to volume marquez_data: wait-for-it.sh
\n
Successfully copied 2.05kB to volumes-provisioner:/db-conf/postgresql.conf
\n
\n
\n
\n", + "lines": 197, + "lines_more": 192, + "preview_is_truncated": true, + "is_starred": false, + "has_rich_preview": false, + "file_access": "visible" + } + ], + "upload": false, + "user": "U05CAULTYG2", + "display_as_bot": false, + "ts": "1698442199.308129", + "client_msg_id": "42d74ed6-31c7-4f4b-b8cd-c61ac213f3d1", + "thread_ts": "1698440472.145489", + "parent_user_id": "U05CAULTYG2" + }, + { + "client_msg_id": "5c3d2cac-67b0-4867-a5f3-f700f2b81a7d", + "type": "message", + "text": "I would first suggest to remove the `--build` flag since you are specifying a version of Marquez to use via `--tag`", + "user": "U01DCMDFHBK", + "ts": "1698442395.550239", + "blocks": [ + { + "type": "rich_text", + "block_id": "+nm/W", + "elements": [ + { + "type": "rich_text_section", + "elements": [ + { + "type": "text", + "text": "I would first suggest to remove the " + }, + { + "type": "text", + "text": "--build", + "style": { + "code": true + } + }, + { + "type": "text", + "text": " flag since you are specifying a version of Marquez to use via " + }, + { + "type": "text", + "text": "--tag", + "style": { + "code": true + } + } + ] + } + ] + } + ], + "team": "T01CWUYP5AR", + "edited": { + "user": "U01DCMDFHBK", + "ts": "1698442406.000000" + }, + "thread_ts": "1698440472.145489", + "parent_user_id": "U05CAULTYG2" + }, + { + "client_msg_id": "def512a3-f14a-4498-a6dd-d8f50ad2452c", + "type": "message", + "text": "no the issue per se, but will help clear up some of the logs", + "user": "U01DCMDFHBK", + "ts": "1698442429.280759", + "blocks": [ + { + "type": "rich_text", + "block_id": "JNwY0", + "elements": [ + { + "type": "rich_text_section", + "elements": [ + { + "type": "text", + "text": "no the issue per se, but will help clear up some of the logs" + } + ] + } + ] + } + ], + "team": "T01CWUYP5AR", + "thread_ts": "1698440472.145489", + "parent_user_id": "U05CAULTYG2" + }, + { + "client_msg_id": "73a94f34-a8b0-4189-966c-6f99563ededd", + "type": "message", + "text": "for sure thanks. we could get the logs without the --build portion, we tried with that option just once", + "user": "U05CAULTYG2", + "ts": "1698442506.399629", + "blocks": [ + { + "type": "rich_text", + "block_id": "0uCEX", + "elements": [ + { + "type": "rich_text_section", + "elements": [ + { + "type": "text", + "text": "for sure thanks. we could get the logs without the --build portion, we tried with that option just once" + } + ] + } + ] + } + ], + "team": "T01CWUYP5AR", + "thread_ts": "1698440472.145489", + "parent_user_id": "U05CAULTYG2" + }, + { + "client_msg_id": "3aec3540-c8ab-440e-8b06-2a5ab2361c8f", + "type": "message", + "text": "the errors were the same with/without --build option", + "user": "U05CAULTYG2", + "ts": "1698442540.055639", + "blocks": [ + { + "type": "rich_text", + "block_id": "qAAA2", + "elements": [ + { + "type": "rich_text_section", + "elements": [ + { + "type": "text", + "text": "the errors were the same with/without --build option" + } + ] + } + ] + } + ], + "team": "T01CWUYP5AR", + "thread_ts": "1698440472.145489", + "parent_user_id": "U05CAULTYG2" + }, + { + "client_msg_id": "dd6c5ac8-a49a-48fb-8416-1ed633ef38a3", + "type": "message", + "text": "marquez-api | ERROR [2023-10-27 21:34:58,019] org.apache.tomcat.jdbc.pool.ConnectionPool: Unable to create initial connections of pool.\n marquez-api | ! org.postgresql.util.PSQLException: FATAL: password authentication failed for user \"marquez\"\n marquez-api | ! at org.postgresql.core.v3.ConnectionFactoryImpl.doAuthentication(ConnectionFactoryImpl.java:693)\n marquez-api | ! at org.postgresql.core.v3.ConnectionFactoryImpl.tryConnect(ConnectionFactoryImpl.java:203)\n marquez-api | ! at org.postgresql.core.v3.ConnectionFactoryImpl.openConnectionImpl(ConnectionFactoryImpl.java:258)\n marquez-api | ! at org.postgresql.core.ConnectionFactory.openConnection(ConnectionFactory.java:54)\n marquez-api | ! at org.postgresql.jdbc.PgConnection.<init>(PgConnection.java:253)\n marquez-api | ! at org.postgresql.Driver.makeConnection(Driver.java:434)\n marquez-api | ! at org.postgresql.Driver.connect(Driver.java:291)\n marquez-api | ! at org.apache.tomcat.jdbc.pool.PooledConnection.connectUsingDriver(PooledConnection.java:346)\n marquez-api | ! at org.apache.tomcat.jdbc.pool.PooledConnection.connect(PooledConnection.java:227)\n marquez-api | ! at org.apache.tomcat.jdbc.pool.ConnectionPool.createConnection(ConnectionPool.java:768)\n marquez-api | ! at org.apache.tomcat.jdbc.pool.ConnectionPool.borrowConnection(ConnectionPool.java:696)\n marquez-api | ! at org.apache.tomcat.jdbc.pool.ConnectionPool.init(ConnectionPool.java:495)\n marquez-api | ! at org.apache.tomcat.jdbc.pool.ConnectionPool.<init>(ConnectionPool.java:153)\n marquez-api | ! at org.apache.tomcat.jdbc.pool.DataSourceProxy.pCreatePool(DataSourceProxy.java:118)\n marquez-api | ! at org.apache.tomcat.jdbc.pool.DataSourceProxy.createPool(DataSourceProxy.java:107)\n marquez-api | ! at org.apache.tomcat.jdbc.pool.DataSourceProxy.getConnection(DataSourceProxy.java:131)\n marquez-api | ! at org.flywaydb.core.internal.jdbc.JdbcUtils.openConnection(JdbcUtils.java:48)\n marquez-api | ! at org.flywaydb.core.internal.jdbc.JdbcConnectionFactory.<init>(JdbcConnectionFactory.java:75)\n marquez-api | ! at org.flywaydb.core.FlywayExecutor.execute(FlywayExecutor.java:147)\n marquez-api | ! at (Flyway.java:190)\n marquez-api | ! at marquez.db.DbMigration.hasPendingDbMigrations(DbMigration.java:73)\n marquez-api | ! at marquez.db.DbMigration.migrateDbOrError(DbMigration.java:27)\n marquez-api | ! at marquez.MarquezApp.run(MarquezApp.java:105)\n marquez-api | ! at marquez.MarquezApp.run(MarquezApp.java:48)\n marquez-api | ! at io.dropwizard.cli.EnvironmentCommand.run(EnvironmentCommand.java:67)\n marquez-api | ! at io.dropwizard.cli.ConfiguredCommand.run(ConfiguredCommand.java:98)\n marquez-api | ! at io.dropwizard.cli.Cli.run(Cli.java:78)\n marquez-api | ! at io.dropwizard.Application.run(Application.java:94)\n marquez-api | ! at marquez.MarquezApp.main(MarquezApp.java:60)\n marquez-api | INFO [2023-10-27 21:34:58,024] marquez.MarquezApp: Stopping app...", + "user": "U05CAULTYG2", + "ts": "1698442562.316779", + "blocks": [ + { + "type": "rich_text", + "block_id": "dzUFO", + "elements": [ + { + "type": "rich_text_section", + "elements": [ + { + "type": "text", + "text": "marquez-api | ERROR [2023-10-27 21:34:58,019] org.apache.tomcat.jdbc.pool.ConnectionPool: Unable to create initial connections of pool.\n marquez-api | ! org.postgresql.util.PSQLException: FATAL: password authentication failed for user \"marquez\"\n marquez-api | ! at org.postgresql.core.v3.ConnectionFactoryImpl.doAuthentication(ConnectionFactoryImpl.java:693)\n marquez-api | ! at org.postgresql.core.v3.ConnectionFactoryImpl.tryConnect(ConnectionFactoryImpl.java:203)\n marquez-api | ! at org.postgresql.core.v3.ConnectionFactoryImpl.openConnectionImpl(ConnectionFactoryImpl.java:258)\n marquez-api | ! at org.postgresql.core.ConnectionFactory.openConnection(ConnectionFactory.java:54)\n marquez-api | ! at org.postgresql.jdbc.PgConnection.(PgConnection.java:253)\n marquez-api | ! at org.postgresql.Driver.makeConnection(Driver.java:434)\n marquez-api | ! at org.postgresql.Driver.connect(Driver.java:291)\n marquez-api | ! at org.apache.tomcat.jdbc.pool.PooledConnection.connectUsingDriver(PooledConnection.java:346)\n marquez-api | ! at org.apache.tomcat.jdbc.pool.PooledConnection.connect(PooledConnection.java:227)\n marquez-api | ! at org.apache.tomcat.jdbc.pool.ConnectionPool.createConnection(ConnectionPool.java:768)\n marquez-api | ! at org.apache.tomcat.jdbc.pool.ConnectionPool.borrowConnection(ConnectionPool.java:696)\n marquez-api | ! at org.apache.tomcat.jdbc.pool.ConnectionPool.init(ConnectionPool.java:495)\n marquez-api | ! at org.apache.tomcat.jdbc.pool.ConnectionPool.(ConnectionPool.java:153)\n marquez-api | ! at org.apache.tomcat.jdbc.pool.DataSourceProxy.pCreatePool(DataSourceProxy.java:118)\n marquez-api | ! at org.apache.tomcat.jdbc.pool.DataSourceProxy.createPool(DataSourceProxy.java:107)\n marquez-api | ! at org.apache.tomcat.jdbc.pool.DataSourceProxy.getConnection(DataSourceProxy.java:131)\n marquez-api | ! at org.flywaydb.core.internal.jdbc.JdbcUtils.openConnection(JdbcUtils.java:48)\n marquez-api | ! at org.flywaydb.core.internal.jdbc.JdbcConnectionFactory.(JdbcConnectionFactory.java:75)\n marquez-api | ! at org.flywaydb.core.FlywayExecutor.execute(FlywayExecutor.java:147)\n marquez-api | ! at " + }, + { + "type": "link", + "url": "http://org.flywaydb.core.Flyway.info", + "text": "org.flywaydb.core.Flyway.info" + }, + { + "type": "text", + "text": "(Flyway.java:190)\n marquez-api | ! at marquez.db.DbMigration.hasPendingDbMigrations(DbMigration.java:73)\n marquez-api | ! at marquez.db.DbMigration.migrateDbOrError(DbMigration.java:27)\n marquez-api | ! at marquez.MarquezApp.run(MarquezApp.java:105)\n marquez-api | ! at marquez.MarquezApp.run(MarquezApp.java:48)\n marquez-api | ! at io.dropwizard.cli.EnvironmentCommand.run(EnvironmentCommand.java:67)\n marquez-api | ! at io.dropwizard.cli.ConfiguredCommand.run(ConfiguredCommand.java:98)\n marquez-api | ! at io.dropwizard.cli.Cli.run(Cli.java:78)\n marquez-api | ! at io.dropwizard.Application.run(Application.java:94)\n marquez-api | ! at marquez.MarquezApp.main(MarquezApp.java:60)\n marquez-api | INFO [2023-10-27 21:34:58,024] marquez.MarquezApp: Stopping app..." + } + ] + } + ] + } + ], + "team": "T01CWUYP5AR", + "thread_ts": "1698440472.145489", + "parent_user_id": "U05CAULTYG2" + }, + { + "client_msg_id": "ce32c470-e688-4743-8a21-9a56adfc16fa", + "type": "message", + "text": "debugging docker issues like this is so difficult", + "user": "U01DCMDFHBK", + "ts": "1698442732.837229", + "blocks": [ + { + "type": "rich_text", + "block_id": "qtDaI", + "elements": [ + { + "type": "rich_text_section", + "elements": [ + { + "type": "text", + "text": "debugging docker issues like this is so difficult" + } + ] + } + ] + } + ], + "team": "T01CWUYP5AR", + "thread_ts": "1698440472.145489", + "parent_user_id": "U05CAULTYG2" + }, + { + "client_msg_id": "19293328-ba28-471f-80cf-e9f3fb13c8a2", + "type": "message", + "text": "it could be a number of things, but you are connected to the database it’s just that the `marquez` user hasn’t been created", + "user": "U01DCMDFHBK", + "ts": "1698442844.009939", + "blocks": [ + { + "type": "rich_text", + "block_id": "+gnXP", + "elements": [ + { + "type": "rich_text_section", + "elements": [ + { + "type": "text", + "text": "it could be a number of things, but you are connected to the database it’s just that the " + }, + { + "type": "text", + "text": "marquez", + "style": { + "code": true + } + }, + { + "type": "text", + "text": " user hasn’t been created" + } + ] + } + ] + } + ], + "team": "T01CWUYP5AR", + "edited": { + "user": "U01DCMDFHBK", + "ts": "1698442854.000000" + }, + "thread_ts": "1698440472.145489", + "parent_user_id": "U05CAULTYG2" + }, + { + "client_msg_id": "84bf8b37-8370-4feb-981d-2ea007a8b564", + "type": "message", + "text": "the is what manages user creation", + "user": "U01DCMDFHBK", + "ts": "1698442919.199779", + "blocks": [ + { + "type": "rich_text", + "block_id": "+hC5M", + "elements": [ + { + "type": "rich_text_section", + "elements": [ + { + "type": "text", + "text": "the " + }, + { + "type": "link", + "url": "https://github.com/MarquezProject/marquez/blob/main/docker/init-db.sh", + "text": "/init-db.sh", + "unsafe": true + }, + { + "type": "text", + "text": " is what manages user creation" + } + ] + } + ] + } + ], + "team": "T01CWUYP5AR", + "edited": { + "user": "U01DCMDFHBK", + "ts": "1698443706.000000" + }, + "thread_ts": "1698440472.145489", + "parent_user_id": "U05CAULTYG2" + }, + { + "client_msg_id": "0e1f65c6-8467-45ff-ae4d-8de0c394366c", + "type": "message", + "text": "so it’s possible that the script isn’t running for whatever reason on your Ec2 instance", + "user": "U01DCMDFHBK", + "ts": "1698442937.126109", + "blocks": [ + { + "type": "rich_text", + "block_id": "Cu5m8", + "elements": [ + { + "type": "rich_text_section", + "elements": [ + { + "type": "text", + "text": "so it’s possible that the script isn’t running for whatever reason on your Ec2 instance" + } + ] + } + ] + } + ], + "team": "T01CWUYP5AR", + "edited": { + "user": "U01DCMDFHBK", + "ts": "1698442952.000000" + }, + "thread_ts": "1698440472.145489", + "parent_user_id": "U05CAULTYG2" + }, + { + "client_msg_id": "64cd0bf6-4183-49fe-8660-5125e06b2837", + "type": "message", + "text": "do you have other services running on that Ec2 instance? Like, other than Marquez", + "user": "U01DCMDFHBK", + "ts": "1698443060.353359", + "blocks": [ + { + "type": "rich_text", + "block_id": "vTxrE", + "elements": [ + { + "type": "rich_text_section", + "elements": [ + { + "type": "text", + "text": "do you have other services running on that Ec2 instance? Like, other than Marquez" + } + ] + } + ] + } + ], + "team": "T01CWUYP5AR", + "thread_ts": "1698440472.145489", + "parent_user_id": "U05CAULTYG2" + }, + { + "client_msg_id": "e13ce2ef-93c0-49f4-b98f-88bf718d1bd5", + "type": "message", + "text": "is there a postgres process running outside of docker?", + "user": "U01DCMDFHBK", + "ts": "1698443092.180609", + "blocks": [ + { + "type": "rich_text", + "block_id": "r88uD", + "elements": [ + { + "type": "rich_text_section", + "elements": [ + { + "type": "text", + "text": "is there a postgres process running outside of docker?" + } + ] + } + ] + } + ], + "team": "T01CWUYP5AR", + "thread_ts": "1698440472.145489", + "parent_user_id": "U05CAULTYG2" + }, + { + "client_msg_id": "ae9a1d48-ddce-4ff3-9d47-f51cc6f96299", + "type": "message", + "text": "no other services except marquez on this EC2 instance", + "user": "U05CAULTYG2", + "ts": "1698453290.531319", + "blocks": [ + { + "type": "rich_text", + "block_id": "ynK45", + "elements": [ + { + "type": "rich_text_section", + "elements": [ + { + "type": "text", + "text": "no other services except marquez on this EC2 instance" + } + ] + } + ] + } + ], + "team": "T01CWUYP5AR", + "thread_ts": "1698440472.145489", + "parent_user_id": "U05CAULTYG2" + }, + { + "client_msg_id": "99164b6f-b536-49cb-94ff-2812cdd776a5", + "type": "message", + "text": "this was a new Ec2 instance that was spun up to install and use marquez", + "user": "U05CAULTYG2", + "ts": "1698453349.420079", + "blocks": [ + { + "type": "rich_text", + "block_id": "GxrIE", + "elements": [ + { + "type": "rich_text_section", + "elements": [ + { + "type": "text", + "text": "this was a new Ec2 instance that was spun up to install and use marquez" + } + ] + } + ] + } + ], + "team": "T01CWUYP5AR", + "thread_ts": "1698440472.145489", + "parent_user_id": "U05CAULTYG2" + }, + { + "client_msg_id": "7643a6c4-a112-4a68-8b8c-84e2e82e757b", + "type": "message", + "text": "n we can confirm that no postgres process runs outside of docker", + "user": "U05CAULTYG2", + "ts": "1698453369.842939", + "blocks": [ + { + "type": "rich_text", + "block_id": "+K3Ga", + "elements": [ + { + "type": "rich_text_section", + "elements": [ + { + "type": "text", + "text": "n we can confirm that no postgres process runs outside of docker" + } + ] + } + ] + } + ], + "team": "T01CWUYP5AR", + "thread_ts": "1698440472.145489", + "parent_user_id": "U05CAULTYG2" + } + ] + }, + { + "type": "message", + "subtype": "thread_broadcast", + "text": "referencing to conversation - what it takes to move to openlineage provider package from openlineage-airflow. Im updating Airflow to 2.7.2 but moving off of openlineage-airflow to provider package Im trying to estimate the amount of work it takes, any thoughts? reading change_logs I dont think its too much of a change but please share your thoughts and if somewhere its drafted please do share that as well", + "user": "U062WLFMRTP", + "ts": "1698429543.349989", + "thread_ts": "1698340358.557159", + "root": { + "type": "message", + "text": "Im upgrading the version from openlineage-airflow==0.24.0 to openlineage-airflow 1.4.1 but im seeing the following error, any help is appreciated", + "files": [ + { + "id": "F062ZFJN2UB", + "created": 1698340299, + "timestamp": 1698340299, + "name": "Screenshot 2023-10-26 at 10.11.34 AM.png", + "title": "Screenshot 2023-10-26 at 10.11.34 AM.png", + "mimetype": "image/png", + "filetype": "png", + "pretty_type": "PNG", + "user": "U062WLFMRTP", + "user_team": "T01CWUYP5AR", + "editable": false, + "size": 356434, + "mode": "hosted", + "is_external": false, + "external_type": "", + "is_public": true, + "public_url_shared": false, + "display_as_bot": false, + "username": "", + "url_private": "https://files.slack.com/files-pri/T01CWUYP5AR-F062ZFJN2UB/screenshot_2023-10-26_at_10.11.34_am.png", + "url_private_download": "https://files.slack.com/files-pri/T01CWUYP5AR-F062ZFJN2UB/download/screenshot_2023-10-26_at_10.11.34_am.png", + "media_display_type": "unknown", + "thumb_64": "https://files.slack.com/files-tmb/T01CWUYP5AR-F062ZFJN2UB-0fca0a48ea/screenshot_2023-10-26_at_10.11.34_am_64.png", + "thumb_80": "https://files.slack.com/files-tmb/T01CWUYP5AR-F062ZFJN2UB-0fca0a48ea/screenshot_2023-10-26_at_10.11.34_am_80.png", + "thumb_360": "https://files.slack.com/files-tmb/T01CWUYP5AR-F062ZFJN2UB-0fca0a48ea/screenshot_2023-10-26_at_10.11.34_am_360.png", + "thumb_360_w": 360, + "thumb_360_h": 222, + "thumb_480": "https://files.slack.com/files-tmb/T01CWUYP5AR-F062ZFJN2UB-0fca0a48ea/screenshot_2023-10-26_at_10.11.34_am_480.png", + "thumb_480_w": 480, + "thumb_480_h": 297, + "thumb_160": "https://files.slack.com/files-tmb/T01CWUYP5AR-F062ZFJN2UB-0fca0a48ea/screenshot_2023-10-26_at_10.11.34_am_160.png", + "thumb_720": "https://files.slack.com/files-tmb/T01CWUYP5AR-F062ZFJN2UB-0fca0a48ea/screenshot_2023-10-26_at_10.11.34_am_720.png", + "thumb_720_w": 720, + "thumb_720_h": 445, + "thumb_800": "https://files.slack.com/files-tmb/T01CWUYP5AR-F062ZFJN2UB-0fca0a48ea/screenshot_2023-10-26_at_10.11.34_am_800.png", + "thumb_800_w": 800, + "thumb_800_h": 494, + "thumb_960": "https://files.slack.com/files-tmb/T01CWUYP5AR-F062ZFJN2UB-0fca0a48ea/screenshot_2023-10-26_at_10.11.34_am_960.png", + "thumb_960_w": 960, + "thumb_960_h": 593, + "thumb_1024": "https://files.slack.com/files-tmb/T01CWUYP5AR-F062ZFJN2UB-0fca0a48ea/screenshot_2023-10-26_at_10.11.34_am_1024.png", + "thumb_1024_w": 1024, + "thumb_1024_h": 633, + "original_w": 1756, + "original_h": 1085, + "thumb_tiny": "AwAdADC8etKKQ9acKADFLRRQAUUUUAMPWnCm96UUAOopM+1GaAFopM0ZoA//2Q==", + "permalink": "https://openlineage.slack.com/files/U062WLFMRTP/F062ZFJN2UB/screenshot_2023-10-26_at_10.11.34_am.png", + "permalink_public": "https://slack-files.com/T01CWUYP5AR-F062ZFJN2UB-c8de1a91b2", + "is_starred": false, + "has_rich_preview": false, + "file_access": "visible" + } + ], + "upload": false, + "user": "U062WLFMRTP", + "display_as_bot": false, + "ts": "1698340358.557159", + "blocks": [ + { + "type": "rich_text", + "block_id": "CRXyh", + "elements": [ + { + "type": "rich_text_section", + "elements": [ + { + "type": "text", + "text": "Im upgrading the version from openlineage-airflow==0.24.0 to openlineage-airflow 1.4.1 but im seeing the following error, any help is appreciated" + } + ] + } + ] + } + ], + "client_msg_id": "925bbcea-663c-4480-8809-2e2b3dd06020", + "thread_ts": "1698340358.557159", + "reply_count": 39, + "reply_users_count": 4, + "latest_reply": "1698786461.272129", + "reply_users": [ + "U062WLFMRTP", + "U02S6F54MAB", + "U01RA9B5GG2", + "U04AZ7992SU" + ], + "is_locked": false, + "subscribed": false + }, + "blocks": [ + { + "type": "rich_text", + "block_id": "bMuXM", + "elements": [ + { + "type": "rich_text_section", + "elements": [ + { + "type": "text", + "text": "referencing to " + }, + { + "type": "link", + "url": "https://openlineage.slack.com/archives/C01CK9T7HKR/p1698398754823079?thread_ts=1698340358.557159&cid=C01CK9T7HKR", + "text": "this" + }, + { + "type": "text", + "text": " conversation - what it takes to move to openlineage provider package from openlineage-airflow. Im updating Airflow to 2.7.2 but moving off of openlineage-airflow to provider package Im trying to estimate the amount of work it takes, any thoughts? reading change_logs I dont think its too much of a change but please share your thoughts and if somewhere its drafted please do share that as well" + } + ] + } + ] + } + ], + "client_msg_id": "0135497a-c1fc-4c07-b449-3c9178cfa2a8" + }, + { + "client_msg_id": "28438eb8-f58b-4cb9-8de5-547e06c2bf90", + "type": "message", + "text": "Hi People, actually I want to intercept the OpenLineage spark events right after the job ends and before they are emitted, so that I can add some extra information to the events or remove some information that I don't want.\nIs there any way of doing this? Can someone please help me", + "user": "U06315TMT61", + "ts": "1698408752.647169", + "blocks": [ + { + "type": "rich_text", + "block_id": "tvUP2", + "elements": [ + { + "type": "rich_text_section", + "elements": [ + { + "type": "text", + "text": "Hi People, actually I want to intercept the OpenLineage spark events right after the job ends and before they are emitted, so that I can add some extra information to the events or remove some information that I don't want.\nIs there any way of doing this? Can someone please help me" + } + ] + } + ] + } + ], + "team": "T01CWUYP5AR", + "thread_ts": "1698408752.647169", + "reply_count": 1, + "reply_users_count": 1, + "latest_reply": "1698671037.466989", + "reply_users": [ + "U02LXF3HUN7" + ], + "is_locked": false, + "subscribed": true, + "last_read": "1698671037.466989", + "replies": [ + { + "client_msg_id": "b4580e02-5543-4906-9c7a-419573392b68", + "type": "message", + "text": "It general, I think this kind of use case is probably best served by , but what do you think <@U02MK6YNAQ5>?", + "user": "U02LXF3HUN7", + "ts": "1698671037.466989", + "blocks": [ + { + "type": "rich_text", + "block_id": "FjMSM", + "elements": [ + { + "type": "rich_text_section", + "elements": [ + { + "type": "text", + "text": "It general, I think this kind of use case is probably best served by " + }, + { + "type": "link", + "url": "https://openlineage.io/docs/guides/facets", + "text": "facets" + }, + { + "type": "text", + "text": ", but what do you think " + }, + { + "type": "user", + "user_id": "U02MK6YNAQ5" + }, + { + "type": "text", + "text": "?" + } + ] + } + ] + } + ], + "team": "T01CWUYP5AR", + "attachments": [ + { + "from_url": "https://openlineage.io/docs/guides/facets", + "service_icon": "https://openlineage.io/img/favicon.ico", + "id": 1, + "original_url": "https://openlineage.io/docs/guides/facets", + "fallback": "Understanding and Using Facets | OpenLineage", + "text": "Adapted from the OpenLineage spec.", + "title": "Understanding and Using Facets | OpenLineage", + "title_link": "https://openlineage.io/docs/guides/facets", + "service_name": "openlineage.io" + } + ], + "thread_ts": "1698408752.647169", + "parent_user_id": "U06315TMT61" + } + ] + }, + { + "client_msg_id": "0683d053-f5f3-4f47-9eda-126d40f8055b", + "type": "message", + "text": "*Spark Integration Logs*\nHey There\nAre these events skipped because it's not supported or it's configured somewhere?\n`23/10/27 08:25:58 INFO SparkSQLExecutionContext: OpenLineage received Spark event that is configured to be skipped: SparkListenerSQLExecutionStart`\n`23/10/27 08:25:58 INFO SparkSQLExecutionContext: OpenLineage received Spark event that is configured to be skipped: SparkListenerSQLExecutionEnd`", + "user": "U05TU0U224A", + "ts": "1698400165.662489", + "blocks": [ + { + "type": "rich_text", + "block_id": "P+NCz", + "elements": [ + { + "type": "rich_text_section", + "elements": [ + { + "type": "text", + "text": "Spark Integration Logs", + "style": { + "bold": true + } + }, + { + "type": "text", + "text": "\nHey There\nAre these events skipped because it's not supported or it's configured somewhere?\n" + }, + { + "type": "text", + "text": "23/10/27 08:25:58 INFO SparkSQLExecutionContext: OpenLineage received Spark event that is configured to be skipped: SparkListenerSQLExecutionStart", + "style": { + "code": true + } + }, + { + "type": "text", + "text": "\n" + }, + { + "type": "text", + "text": "23/10/27 08:25:58 INFO SparkSQLExecutionContext: OpenLineage received Spark event that is configured to be skipped: SparkListenerSQLExecutionEnd", + "style": { + "code": true + } + } + ] + } + ] + } + ], + "team": "T01CWUYP5AR", + "edited": { + "user": "U05TU0U224A", + "ts": "1698400172.000000" + } + }, + { + "type": "message", + "text": "Im upgrading the version from openlineage-airflow==0.24.0 to openlineage-airflow 1.4.1 but im seeing the following error, any help is appreciated", + "files": [ + { + "id": "F062ZFJN2UB", + "created": 1698340299, + "timestamp": 1698340299, + "name": "Screenshot 2023-10-26 at 10.11.34 AM.png", + "title": "Screenshot 2023-10-26 at 10.11.34 AM.png", + "mimetype": "image/png", + "filetype": "png", + "pretty_type": "PNG", + "user": "U062WLFMRTP", + "user_team": "T01CWUYP5AR", + "editable": false, + "size": 356434, + "mode": "hosted", + "is_external": false, + "external_type": "", + "is_public": true, + "public_url_shared": false, + "display_as_bot": false, + "username": "", + "url_private": "https://files.slack.com/files-pri/T01CWUYP5AR-F062ZFJN2UB/screenshot_2023-10-26_at_10.11.34_am.png", + "url_private_download": "https://files.slack.com/files-pri/T01CWUYP5AR-F062ZFJN2UB/download/screenshot_2023-10-26_at_10.11.34_am.png", + "media_display_type": "unknown", + "thumb_64": "https://files.slack.com/files-tmb/T01CWUYP5AR-F062ZFJN2UB-0fca0a48ea/screenshot_2023-10-26_at_10.11.34_am_64.png", + "thumb_80": "https://files.slack.com/files-tmb/T01CWUYP5AR-F062ZFJN2UB-0fca0a48ea/screenshot_2023-10-26_at_10.11.34_am_80.png", + "thumb_360": "https://files.slack.com/files-tmb/T01CWUYP5AR-F062ZFJN2UB-0fca0a48ea/screenshot_2023-10-26_at_10.11.34_am_360.png", + "thumb_360_w": 360, + "thumb_360_h": 222, + "thumb_480": "https://files.slack.com/files-tmb/T01CWUYP5AR-F062ZFJN2UB-0fca0a48ea/screenshot_2023-10-26_at_10.11.34_am_480.png", + "thumb_480_w": 480, + "thumb_480_h": 297, + "thumb_160": "https://files.slack.com/files-tmb/T01CWUYP5AR-F062ZFJN2UB-0fca0a48ea/screenshot_2023-10-26_at_10.11.34_am_160.png", + "thumb_720": "https://files.slack.com/files-tmb/T01CWUYP5AR-F062ZFJN2UB-0fca0a48ea/screenshot_2023-10-26_at_10.11.34_am_720.png", + "thumb_720_w": 720, + "thumb_720_h": 445, + "thumb_800": "https://files.slack.com/files-tmb/T01CWUYP5AR-F062ZFJN2UB-0fca0a48ea/screenshot_2023-10-26_at_10.11.34_am_800.png", + "thumb_800_w": 800, + "thumb_800_h": 494, + "thumb_960": "https://files.slack.com/files-tmb/T01CWUYP5AR-F062ZFJN2UB-0fca0a48ea/screenshot_2023-10-26_at_10.11.34_am_960.png", + "thumb_960_w": 960, + "thumb_960_h": 593, + "thumb_1024": "https://files.slack.com/files-tmb/T01CWUYP5AR-F062ZFJN2UB-0fca0a48ea/screenshot_2023-10-26_at_10.11.34_am_1024.png", + "thumb_1024_w": 1024, + "thumb_1024_h": 633, + "original_w": 1756, + "original_h": 1085, + "thumb_tiny": "AwAdADC8etKKQ9acKADFLRRQAUUUUAMPWnCm96UUAOopM+1GaAFopM0ZoA//2Q==", + "permalink": "https://openlineage.slack.com/files/U062WLFMRTP/F062ZFJN2UB/screenshot_2023-10-26_at_10.11.34_am.png", + "permalink_public": "https://slack-files.com/T01CWUYP5AR-F062ZFJN2UB-c8de1a91b2", + "is_starred": false, + "has_rich_preview": false, + "file_access": "visible" + } + ], + "upload": false, + "user": "U062WLFMRTP", + "display_as_bot": false, + "ts": "1698340358.557159", + "blocks": [ + { + "type": "rich_text", + "block_id": "CRXyh", + "elements": [ + { + "type": "rich_text_section", + "elements": [ + { + "type": "text", + "text": "Im upgrading the version from openlineage-airflow==0.24.0 to openlineage-airflow 1.4.1 but im seeing the following error, any help is appreciated" + } + ] + } + ] + } + ], + "client_msg_id": "925bbcea-663c-4480-8809-2e2b3dd06020", + "thread_ts": "1698340358.557159", + "reply_count": 39, + "reply_users_count": 4, + "latest_reply": "1698786461.272129", + "reply_users": [ + "U062WLFMRTP", + "U02S6F54MAB", + "U01RA9B5GG2", + "U04AZ7992SU" + ], + "is_locked": false, + "subscribed": false, + "replies": [ + { + "client_msg_id": "36c27c05-57eb-4ff9-8a1b-a4e0b4e7c5ee", + "type": "message", + "text": "<@U02S6F54MAB> any thoughts?", + "user": "U062WLFMRTP", + "ts": "1698340442.113079", + "blocks": [ + { + "type": "rich_text", + "block_id": "Zq5Bz", + "elements": [ + { + "type": "rich_text_section", + "elements": [ + { + "type": "user", + "user_id": "U02S6F54MAB" + }, + { + "type": "text", + "text": " any thoughts?" + } + ] + } + ] + } + ], + "team": "T01CWUYP5AR", + "thread_ts": "1698340358.557159", + "parent_user_id": "U062WLFMRTP" + }, + { + "client_msg_id": "d35a745b-327a-495b-a182-948c7012399a", + "type": "message", + "text": "what version of Airflow are you using?", + "user": "U02S6F54MAB", + "ts": "1698340464.257599", + "blocks": [ + { + "type": "rich_text", + "block_id": "Dheg7", + "elements": [ + { + "type": "rich_text_section", + "elements": [ + { + "type": "text", + "text": "what version of Airflow are you using?" + } + ] + } + ] + } + ], + "team": "T01CWUYP5AR", + "thread_ts": "1698340358.557159", + "parent_user_id": "U062WLFMRTP" + }, + { + "client_msg_id": "0fb8b18d-009d-45c6-81e1-8bea9a4e1436", + "type": "message", + "text": "2.6.3 that satisfies the requirement", + "user": "U062WLFMRTP", + "ts": "1698340492.528189", + "blocks": [ + { + "type": "rich_text", + "block_id": "vUCUN", + "elements": [ + { + "type": "rich_text_section", + "elements": [ + { + "type": "text", + "text": "2.6.3 that satisfies the requirement" + } + ] + } + ] + } + ], + "team": "T01CWUYP5AR", + "thread_ts": "1698340358.557159", + "parent_user_id": "U062WLFMRTP" + }, + { + "client_msg_id": "14706124-4868-4b05-9015-ec735e92f234", + "type": "message", + "text": "is it possible you have some custom operator?", + "user": "U02S6F54MAB", + "ts": "1698340598.078979", + "blocks": [ + { + "type": "rich_text", + "block_id": "vK+L0", + "elements": [ + { + "type": "rich_text_section", + "elements": [ + { + "type": "text", + "text": "is it possible you have some custom operator?" + } + ] + } + ] + } + ], + "team": "T01CWUYP5AR", + "thread_ts": "1698340358.557159", + "parent_user_id": "U062WLFMRTP" + }, + { + "client_msg_id": "396e6c98-d994-4805-aeba-82ceb7aeb7eb", + "type": "message", + "text": "i think its the base operator causing the issue", + "user": "U062WLFMRTP", + "ts": "1698340635.640199", + "blocks": [ + { + "type": "rich_text", + "block_id": "mcI4u", + "elements": [ + { + "type": "rich_text_section", + "elements": [ + { + "type": "text", + "text": "i think its the base operator causing the issue" + } + ] + } + ] + } + ], + "team": "T01CWUYP5AR", + "thread_ts": "1698340358.557159", + "parent_user_id": "U062WLFMRTP" + }, + { + "client_msg_id": "ab03eaa3-7620-4f81-b8fb-950db728e114", + "type": "message", + "text": "so no i believe", + "user": "U062WLFMRTP", + "ts": "1698340656.239529", + "blocks": [ + { + "type": "rich_text", + "block_id": "lqOsV", + "elements": [ + { + "type": "rich_text_section", + "elements": [ + { + "type": "text", + "text": "so no i believe" + } + ] + } + ] + } + ], + "team": "T01CWUYP5AR", + "thread_ts": "1698340358.557159", + "parent_user_id": "U062WLFMRTP" + }, + { + "client_msg_id": "786dbec2-66c6-4761-9d00-23401ba14a0a", + "type": "message", + "text": "BaseOperator is parent class for any other operators, it defines how to do deepcopy", + "user": "U02S6F54MAB", + "ts": "1698340723.317029", + "blocks": [ + { + "type": "rich_text", + "block_id": "jre16", + "elements": [ + { + "type": "rich_text_section", + "elements": [ + { + "type": "text", + "text": "BaseOperator is parent class for any other operators, it defines how to do deepcopy" + } + ] + } + ] + } + ], + "team": "T01CWUYP5AR", + "thread_ts": "1698340358.557159", + "parent_user_id": "U062WLFMRTP" + }, + { + "client_msg_id": "09433b61-e80a-40b0-8c50-30a67307ed11", + "type": "message", + "text": "yeah so its controlled by Airflow itself, I didnt customize it", + "user": "U062WLFMRTP", + "ts": "1698340751.496409", + "blocks": [ + { + "type": "rich_text", + "block_id": "Gi0r4", + "elements": [ + { + "type": "rich_text_section", + "elements": [ + { + "type": "text", + "text": "yeah so its controlled by Airflow itself, I didnt customize it" + } + ] + } + ] + } + ], + "team": "T01CWUYP5AR", + "thread_ts": "1698340358.557159", + "parent_user_id": "U062WLFMRTP" + }, + { + "client_msg_id": "640ee14f-9771-46d2-a64a-926341256843", + "type": "message", + "text": "uhm, maybe it's possible you could share dag code? you may hide sensitive data", + "user": "U02S6F54MAB", + "ts": "1698340789.650199", + "blocks": [ + { + "type": "rich_text", + "block_id": "q/Vb4", + "elements": [ + { + "type": "rich_text_section", + "elements": [ + { + "type": "text", + "text": "uhm, maybe it's possible you could share dag code? you may hide sensitive data" + } + ] + } + ] + } + ], + "team": "T01CWUYP5AR", + "thread_ts": "1698340358.557159", + "parent_user_id": "U062WLFMRTP" + }, + { + "client_msg_id": "207201cb-bf3f-415c-851c-8ba731be623d", + "type": "message", + "text": "let me try with lower versions of openlineage, what's say", + "user": "U062WLFMRTP", + "ts": "1698340883.515319", + "blocks": [ + { + "type": "rich_text", + "block_id": "uhz4a", + "elements": [ + { + "type": "rich_text_section", + "elements": [ + { + "type": "text", + "text": "let me try with lower versions of openlineage, what's say" + } + ] + } + ] + } + ], + "team": "T01CWUYP5AR", + "thread_ts": "1698340358.557159", + "parent_user_id": "U062WLFMRTP" + }, + { + "client_msg_id": "79d09ca2-28ae-49cc-a2a2-6d8d0ff4665e", + "type": "message", + "text": "its a big jump from 0.24.0 to 1.4.1", + "user": "U062WLFMRTP", + "ts": "1698340899.088059", + "blocks": [ + { + "type": "rich_text", + "block_id": "bOtmE", + "elements": [ + { + "type": "rich_text_section", + "elements": [ + { + "type": "text", + "text": "its a big jump from 0.24.0 to 1.4.1" + } + ] + } + ] + } + ], + "team": "T01CWUYP5AR", + "thread_ts": "1698340358.557159", + "parent_user_id": "U062WLFMRTP" + }, + { + "client_msg_id": "01860990-c896-44c9-b9d0-525f56827276", + "type": "message", + "text": "but i will help here to investigate this issue", + "user": "U062WLFMRTP", + "ts": "1698340945.409289", + "blocks": [ + { + "type": "rich_text", + "block_id": "s22lz", + "elements": [ + { + "type": "rich_text_section", + "elements": [ + { + "type": "text", + "text": "but i will help here to investigate this issue" + } + ] + } + ] + } + ], + "team": "T01CWUYP5AR", + "thread_ts": "1698340358.557159", + "parent_user_id": "U062WLFMRTP" + }, + { + "client_msg_id": "9ac38164-dc76-4a5c-9cb3-69dd9eb5fd67", + "type": "message", + "text": "for me it seems that within dag or task you're defining some object that is not easy to copy", + "user": "U02S6F54MAB", + "ts": "1698341043.978079", + "blocks": [ + { + "type": "rich_text", + "block_id": "eWHBd", + "elements": [ + { + "type": "rich_text_section", + "elements": [ + { + "type": "text", + "text": "for me it seems that within dag or task you're defining some object that is not easy to copy" + } + ] + } + ] + } + ], + "team": "T01CWUYP5AR", + "thread_ts": "1698340358.557159", + "parent_user_id": "U062WLFMRTP" + }, + { + "client_msg_id": "1cc0e144-e224-4edc-adcb-25418ccadaa6", + "type": "message", + "text": "possible, but with 0.24.0 that issue is not occurring, so worry is that the version upgrade could potentially break things", + "user": "U062WLFMRTP", + "ts": "1698341165.007029", + "blocks": [ + { + "type": "rich_text", + "block_id": "vpwe5", + "elements": [ + { + "type": "rich_text_section", + "elements": [ + { + "type": "text", + "text": "possible, but with 0.24.0 that issue is not occurring, so worry is that the version upgrade could potentially break things" + } + ] + } + ] + } + ], + "team": "T01CWUYP5AR", + "thread_ts": "1698340358.557159", + "parent_user_id": "U062WLFMRTP" + }, + { + "client_msg_id": "a805ad1c-218e-4d80-8265-5a75094fc00d", + "type": "message", + "text": "0.24.0 is not that old :thinking_face:", + "user": "U02S6F54MAB", + "ts": "1698341974.726729", + "blocks": [ + { + "type": "rich_text", + "block_id": "PhAvj", + "elements": [ + { + "type": "rich_text_section", + "elements": [ + { + "type": "text", + "text": "0.24.0 is not that old " + }, + { + "type": "emoji", + "name": "thinking_face", + "unicode": "1f914" + } + ] + } + ] + } + ], + "team": "T01CWUYP5AR", + "thread_ts": "1698340358.557159", + "parent_user_id": "U062WLFMRTP" + }, + { + "client_msg_id": "add426a1-3614-4a27-8fe3-bbc3939d0a6f", + "type": "message", + "text": "i see the issue with 0.24.0 I see it as warning\n```[airflow/utils/log/logging_mixin.py::_propagate_log()::150] WARNING - File \"/usr/lib64/python3.8/threading.py\", line 932, in _bootstrap_inner\n[2023-10-26, 17:40:50 UTC] [airflow/utils/log/logging_mixin.py::_propagate_log()::150] WARNING - self.run()\n[2023-10-26, 17:40:50 UTC] [airflow/utils/log/logging_mixin.py::_propagate_log()::150] WARNING - File \"/usr/lib64/python3.8/threading.py\", line 870, in run\n[2023-10-26, 17:40:50 UTC] [airflow/utils/log/logging_mixin.py::_propagate_log()::150] WARNING - self._target(*self._args, **self._kwargs)\n[2023-10-26, 17:40:50 UTC] [airflow/utils/log/logging_mixin.py::_propagate_log()::150] WARNING - File \"/home/upgrade/.local/lib/python3.8/site-packages/openlineage/airflow/listener.py\", line 89, in on_running\n[2023-10-26, 17:40:50 UTC] [airflow/utils/log/logging_mixin.py::_propagate_log()::150] WARNING - task_instance_copy = copy.deepcopy(task_instance)\n[2023-10-26, 17:40:50 UTC] [airflow/utils/log/logging_mixin.py::_propagate_log()::150] WARNING - File \"/usr/lib64/python3.8/copy.py\", line 172, in deepcopy\n[2023-10-26, 17:40:50 UTC] [airflow/utils/log/logging_mixin.py::_propagate_log()::150] WARNING - y = _reconstruct(x, memo, *rv)\n[2023-10-26, 17:40:50 UTC] [airflow/utils/log/logging_mixin.py::_propagate_log()::150] WARNING - File \"/usr/lib64/python3.8/copy.py\", line 270, in _reconstruct\n[2023-10-26, 17:40:50 UTC] [airflow/utils/log/logging_mixin.py::_propagate_log()::150] WARNING - state = deepcopy(state, memo)\n[2023-10-26, 17:40:50 UTC] [airflow/utils/log/logging_mixin.py::_propagate_log()::150] WARNING - File \"/usr/lib64/python3.8/copy.py\", line 146, in deepcopy\n[2023-10-26, 17:40:50 UTC] [airflow/utils/log/logging_mixin.py::_propagate_log()::150] WARNING - y = copier(x, memo)\n[2023-10-26, 17:40:50 UTC] [airflow/utils/log/logging_mixin.py::_propagate_log()::150] WARNING - File \"/usr/lib64/python3.8/copy.py\", line 230, in _deepcopy_dict\n[2023-10-26, 17:40:50 UTC] [airflow/utils/log/logging_mixin.py::_propagate_log()::150] WARNING - y[deepcopy(key, memo)] = deepcopy(value, memo)\n[2023-10-26, 17:40:50 UTC] [airflow/utils/log/logging_mixin.py::_propagate_log()::150] WARNING - File \"/usr/lib64/python3.8/copy.py\", line 172, in deepcopy\n[2023-10-26, 17:40:50 UTC] [airflow/utils/log/logging_mixin.py::_propagate_log()::150] WARNING - y = _reconstruct(x, memo, *rv)\n[2023-10-26, 17:40:50 UTC] [airflow/utils/log/logging_mixin.py::_propagate_log()::150] WARNING - File \"/usr/lib64/python3.8/copy.py\", line 270, in _reconstruct\n[2023-10-26, 17:40:50 UTC] [airflow/utils/log/logging_mixin.py::_propagate_log()::150] WARNING - state = deepcopy(state, memo)\n[2023-10-26, 17:40:50 UTC] [airflow/utils/log/logging_mixin.py::_propagate_log()::150] WARNING - File \"/usr/lib64/python3.8/copy.py\", line 146, in deepcopy\n[2023-10-26, 17:40:50 UTC] [airflow/utils/log/logging_mixin.py::_propagate_log()::150] WARNING - y = copier(x, memo)\n[2023-10-26, 17:40:50 UTC] [airflow/utils/log/logging_mixin.py::_propagate_log()::150] WARNING - File \"/usr/lib64/python3.8/copy.py\", line 230, in _deepcopy_dict\n[2023-10-26, 17:40:50 UTC] [airflow/utils/log/logging_mixin.py::_propagate_log()::150] WARNING - y[deepcopy(key, memo)] = deepcopy(value, memo)\n[2023-10-26, 17:40:50 UTC] [airflow/utils/log/logging_mixin.py::_propagate_log()::150] WARNING - File \"/usr/lib64/python3.8/copy.py\", line 153, in deepcopy\n[2023-10-26, 17:40:50 UTC] [airflow/utils/log/logging_mixin.py::_propagate_log()::150] WARNING - y = copier(memo)\n[2023-10-26, 17:40:50 UTC] [airflow/utils/log/logging_mixin.py::_propagate_log()::150] WARNING - File \"/home/upgrade/.local/lib/python3.8/site-packages/airflow/models/dag.py\", line 2162, in __deepcopy__\n[2023-10-26, 17:40:50 UTC] [airflow/utils/log/logging_mixin.py::_propagate_log()::150] WARNING - setattr(result, k, copy.deepcopy(v, memo))\n[2023-10-26, 17:40:50 UTC] [airflow/utils/log/logging_mixin.py::_propagate_log()::150] WARNING - File \"/usr/lib64/python3.8/copy.py\", line 146, in deepcopy\n[2023-10-26, 17:40:50 UTC] [airflow/utils/log/logging_mixin.py::_propagate_log()::150] WARNING - y = copier(x, memo)\n[2023-10-26, 17:40:50 UTC] [airflow/utils/log/logging_mixin.py::_propagate_log()::150] WARNING - File \"/usr/lib64/python3.8/copy.py\", line 230, in _deepcopy_dict\n[2023-10-26, 17:40:50 UTC] [airflow/utils/log/logging_mixin.py::_propagate_log()::150] WARNING - y[deepcopy(key, memo)] = deepcopy(value, memo)\n[2023-10-26, 17:40:50 UTC] [airflow/utils/log/logging_mixin.py::_propagate_log()::150] WARNING - File \"/usr/lib64/python3.8/copy.py\", line 153, in deepcopy\n[2023-10-26, 17:40:50 UTC] [airflow/utils/log/logging_mixin.py::_propagate_log()::150] WARNING - y = copier(memo)\n[2023-10-26, 17:40:50 UTC] [airflow/utils/log/logging_mixin.py::_propagate_log()::150] WARNING - File \"/home/upgrade/.local/lib/python3.8/site-packages/airflow/models/baseoperator.py\", line 1224, in __deepcopy__\n[2023-10-26, 17:40:50 UTC] [airflow/utils/log/logging_mixin.py::_propagate_log()::150] WARNING - setattr(result, k, copy.deepcopy(v, memo))\n[2023-10-26, 17:40:50 UTC] [airflow/utils/log/logging_mixin.py::_propagate_log()::150] WARNING - File \"/usr/lib64/python3.8/copy.py\", line 172, in deepcopy\n[2023-10-26, 17:40:50 UTC] [airflow/utils/log/logging_mixin.py::_propagate_log()::150] WARNING - y = _reconstruct(x, memo, *rv)\n[2023-10-26, 17:40:50 UTC] [airflow/utils/log/logging_mixin.py::_propagate_log()::150] WARNING - File \"/usr/lib64/python3.8/copy.py\", line 270, in _reconstruct\n[2023-10-26, 17:40:50 UTC] [airflow/utils/log/logging_mixin.py::_propagate_log()::150] WARNING - state = deepcopy(state, memo)\n[2023-10-26, 17:40:50 UTC] [airflow/utils/log/logging_mixin.py::_propagate_log()::150] WARNING - File \"/usr/lib64/python3.8/copy.py\", line 146, in deepcopy\n[2023-10-26, 17:40:50 UTC] [airflow/utils/log/logging_mixin.py::_propagate_log()::150] WARNING - y = copier(x, memo)\n[2023-10-26, 17:40:50 UTC] [airflow/utils/log/logging_mixin.py::_propagate_log()::150] WARNING - File \"/usr/lib64/python3.8/copy.py\", line 230, in _deepcopy_dict\n[2023-10-26, 17:40:50 UTC] [airflow/utils/log/logging_mixin.py::_propagate_log()::150] WARNING - y[deepcopy(key, memo)] = deepcopy(value, memo)\n[2023-10-26, 17:40:50 UTC] [airflow/utils/log/logging_mixin.py::_propagate_log()::150] WARNING - File \"/usr/lib64/python3.8/copy.py\", line 146, in deepcopy\n[2023-10-26, 17:40:50 UTC] [airflow/utils/log/logging_mixin.py::_propagate_log()::150] WARNING - y = copier(x, memo)\n[2023-10-26, 17:40:50 UTC] [airflow/utils/log/logging_mixin.py::_propagate_log()::150] WARNING - File \"/usr/lib64/python3.8/copy.py\", line 230, in _deepcopy_dict\n[2023-10-26, 17:40:50 UTC] [airflow/utils/log/logging_mixin.py::_propagate_log()::150] WARNING - y[deepcopy(key, memo)] = deepcopy(value, memo)\n[2023-10-26, 17:40:50 UTC] [airflow/utils/log/logging_mixin.py::_propagate_log()::150] WARNING - File \"/usr/lib64/python3.8/copy.py\", line 153, in deepcopy\n[2023-10-26, 17:40:50 UTC] [airflow/utils/log/logging_mixin.py::_propagate_log()::150] WARNING - y = copier(memo)\n[2023-10-26, 17:40:50 UTC] [airflow/utils/log/logging_mixin.py::_propagate_log()::150] WARNING - File \"/home/upgrade/.local/lib/python3.8/site-packages/airflow/models/baseoperator.py\", line 1224, in __deepcopy__\n[2023-10-26, 17:40:50 UTC] [airflow/utils/log/logging_mixin.py::_propagate_log()::150] WARNING - setattr(result, k, copy.deepcopy(v, memo))\n[2023-10-26, 17:40:50 UTC] [airflow/utils/log/logging_mixin.py::_propagate_log()::150] WARNING - File \"/usr/lib64/python3.8/copy.py\", line 146, in deepcopy\n[2023-10-26, 17:40:50 UTC] [airflow/utils/log/logging_mixin.py::_propagate_log()::150] WARNING - y = copier(x, memo)\n[2023-10-26, 17:40:50 UTC] [airflow/utils/log/logging_mixin.py::_propagate_log()::150] WARNING - File \"/usr/lib64/python3.8/copy.py\", line 230, in _deepcopy_dict\n[2023-10-26, 17:40:50 UTC] [airflow/utils/log/logging_mixin.py::_propagate_log()::150] WARNING - y[deepcopy(key, memo)] = deepcopy(value, memo)\n[2023-10-26, 17:40:50 UTC] [airflow/utils/log/logging_mixin.py::_propagate_log()::150] WARNING - File \"/usr/lib64/python3.8/copy.py\", line 161, in deepcopy\n[2023-10-26, 17:40:50 UTC] [airflow/utils/log/logging_mixin.py::_propagate_log()::150] WARNING - rv = reductor(4)\n[2023-10-26, 17:40:50 UTC] [airflow/utils/log/logging_mixin.py::_propagate_log()::150] WARNING - TypeError: cannot pickle 'module' object```\nbut with 1.4.1 its stopped processing any further and threw error", + "user": "U062WLFMRTP", + "ts": "1698342307.308619", + "blocks": [ + { + "type": "rich_text", + "block_id": "AT+nJ", + "elements": [ + { + "type": "rich_text_section", + "elements": [ + { + "type": "text", + "text": "i see the issue with 0.24.0 I see it as warning\n" + } + ] + }, + { + "type": "rich_text_preformatted", + "elements": [ + { + "type": "text", + "text": "[airflow/utils/log/logging_mixin.py::_propagate_log()::150] WARNING - File \"/usr/lib64/python3.8/threading.py\", line 932, in _bootstrap_inner\n[2023-10-26, 17:40:50 UTC] [airflow/utils/log/logging_mixin.py::_propagate_log()::150] WARNING - self.run()\n[2023-10-26, 17:40:50 UTC] [airflow/utils/log/logging_mixin.py::_propagate_log()::150] WARNING - File \"/usr/lib64/python3.8/threading.py\", line 870, in run\n[2023-10-26, 17:40:50 UTC] [airflow/utils/log/logging_mixin.py::_propagate_log()::150] WARNING - self._target(*self._args, **self._kwargs)\n[2023-10-26, 17:40:50 UTC] [airflow/utils/log/logging_mixin.py::_propagate_log()::150] WARNING - File \"/home/upgrade/.local/lib/python3.8/site-packages/openlineage/airflow/listener.py\", line 89, in on_running\n[2023-10-26, 17:40:50 UTC] [airflow/utils/log/logging_mixin.py::_propagate_log()::150] WARNING - task_instance_copy = copy.deepcopy(task_instance)\n[2023-10-26, 17:40:50 UTC] [airflow/utils/log/logging_mixin.py::_propagate_log()::150] WARNING - File \"/usr/lib64/python3.8/copy.py\", line 172, in deepcopy\n[2023-10-26, 17:40:50 UTC] [airflow/utils/log/logging_mixin.py::_propagate_log()::150] WARNING - y = _reconstruct(x, memo, *rv)\n[2023-10-26, 17:40:50 UTC] [airflow/utils/log/logging_mixin.py::_propagate_log()::150] WARNING - File \"/usr/lib64/python3.8/copy.py\", line 270, in _reconstruct\n[2023-10-26, 17:40:50 UTC] [airflow/utils/log/logging_mixin.py::_propagate_log()::150] WARNING - state = deepcopy(state, memo)\n[2023-10-26, 17:40:50 UTC] [airflow/utils/log/logging_mixin.py::_propagate_log()::150] WARNING - File \"/usr/lib64/python3.8/copy.py\", line 146, in deepcopy\n[2023-10-26, 17:40:50 UTC] [airflow/utils/log/logging_mixin.py::_propagate_log()::150] WARNING - y = copier(x, memo)\n[2023-10-26, 17:40:50 UTC] [airflow/utils/log/logging_mixin.py::_propagate_log()::150] WARNING - File \"/usr/lib64/python3.8/copy.py\", line 230, in _deepcopy_dict\n[2023-10-26, 17:40:50 UTC] [airflow/utils/log/logging_mixin.py::_propagate_log()::150] WARNING - y[deepcopy(key, memo)] = deepcopy(value, memo)\n[2023-10-26, 17:40:50 UTC] [airflow/utils/log/logging_mixin.py::_propagate_log()::150] WARNING - File \"/usr/lib64/python3.8/copy.py\", line 172, in deepcopy\n[2023-10-26, 17:40:50 UTC] [airflow/utils/log/logging_mixin.py::_propagate_log()::150] WARNING - y = _reconstruct(x, memo, *rv)\n[2023-10-26, 17:40:50 UTC] [airflow/utils/log/logging_mixin.py::_propagate_log()::150] WARNING - File \"/usr/lib64/python3.8/copy.py\", line 270, in _reconstruct\n[2023-10-26, 17:40:50 UTC] [airflow/utils/log/logging_mixin.py::_propagate_log()::150] WARNING - state = deepcopy(state, memo)\n[2023-10-26, 17:40:50 UTC] [airflow/utils/log/logging_mixin.py::_propagate_log()::150] WARNING - File \"/usr/lib64/python3.8/copy.py\", line 146, in deepcopy\n[2023-10-26, 17:40:50 UTC] [airflow/utils/log/logging_mixin.py::_propagate_log()::150] WARNING - y = copier(x, memo)\n[2023-10-26, 17:40:50 UTC] [airflow/utils/log/logging_mixin.py::_propagate_log()::150] WARNING - File \"/usr/lib64/python3.8/copy.py\", line 230, in _deepcopy_dict\n[2023-10-26, 17:40:50 UTC] [airflow/utils/log/logging_mixin.py::_propagate_log()::150] WARNING - y[deepcopy(key, memo)] = deepcopy(value, memo)\n[2023-10-26, 17:40:50 UTC] [airflow/utils/log/logging_mixin.py::_propagate_log()::150] WARNING - File \"/usr/lib64/python3.8/copy.py\", line 153, in deepcopy\n[2023-10-26, 17:40:50 UTC] [airflow/utils/log/logging_mixin.py::_propagate_log()::150] WARNING - y = copier(memo)\n[2023-10-26, 17:40:50 UTC] [airflow/utils/log/logging_mixin.py::_propagate_log()::150] WARNING - File \"/home/upgrade/.local/lib/python3.8/site-packages/airflow/models/dag.py\", line 2162, in __deepcopy__\n[2023-10-26, 17:40:50 UTC] [airflow/utils/log/logging_mixin.py::_propagate_log()::150] WARNING - setattr(result, k, copy.deepcopy(v, memo))\n[2023-10-26, 17:40:50 UTC] [airflow/utils/log/logging_mixin.py::_propagate_log()::150] WARNING - File \"/usr/lib64/python3.8/copy.py\", line 146, in deepcopy\n[2023-10-26, 17:40:50 UTC] [airflow/utils/log/logging_mixin.py::_propagate_log()::150] WARNING - y = copier(x, memo)\n[2023-10-26, 17:40:50 UTC] [airflow/utils/log/logging_mixin.py::_propagate_log()::150] WARNING - File \"/usr/lib64/python3.8/copy.py\", line 230, in _deepcopy_dict\n[2023-10-26, 17:40:50 UTC] [airflow/utils/log/logging_mixin.py::_propagate_log()::150] WARNING - y[deepcopy(key, memo)] = deepcopy(value, memo)\n[2023-10-26, 17:40:50 UTC] [airflow/utils/log/logging_mixin.py::_propagate_log()::150] WARNING - File \"/usr/lib64/python3.8/copy.py\", line 153, in deepcopy\n[2023-10-26, 17:40:50 UTC] [airflow/utils/log/logging_mixin.py::_propagate_log()::150] WARNING - y = copier(memo)\n[2023-10-26, 17:40:50 UTC] [airflow/utils/log/logging_mixin.py::_propagate_log()::150] WARNING - File \"/home/upgrade/.local/lib/python3.8/site-packages/airflow/models/baseoperator.py\", line 1224, in __deepcopy__\n[2023-10-26, 17:40:50 UTC] [airflow/utils/log/logging_mixin.py::_propagate_log()::150] WARNING - setattr(result, k, copy.deepcopy(v, memo))\n[2023-10-26, 17:40:50 UTC] [airflow/utils/log/logging_mixin.py::_propagate_log()::150] WARNING - File \"/usr/lib64/python3.8/copy.py\", line 172, in deepcopy\n[2023-10-26, 17:40:50 UTC] [airflow/utils/log/logging_mixin.py::_propagate_log()::150] WARNING - y = _reconstruct(x, memo, *rv)\n[2023-10-26, 17:40:50 UTC] [airflow/utils/log/logging_mixin.py::_propagate_log()::150] WARNING - File \"/usr/lib64/python3.8/copy.py\", line 270, in _reconstruct\n[2023-10-26, 17:40:50 UTC] [airflow/utils/log/logging_mixin.py::_propagate_log()::150] WARNING - state = deepcopy(state, memo)\n[2023-10-26, 17:40:50 UTC] [airflow/utils/log/logging_mixin.py::_propagate_log()::150] WARNING - File \"/usr/lib64/python3.8/copy.py\", line 146, in deepcopy\n[2023-10-26, 17:40:50 UTC] [airflow/utils/log/logging_mixin.py::_propagate_log()::150] WARNING - y = copier(x, memo)\n[2023-10-26, 17:40:50 UTC] [airflow/utils/log/logging_mixin.py::_propagate_log()::150] WARNING - File \"/usr/lib64/python3.8/copy.py\", line 230, in _deepcopy_dict\n[2023-10-26, 17:40:50 UTC] [airflow/utils/log/logging_mixin.py::_propagate_log()::150] WARNING - y[deepcopy(key, memo)] = deepcopy(value, memo)\n[2023-10-26, 17:40:50 UTC] [airflow/utils/log/logging_mixin.py::_propagate_log()::150] WARNING - File \"/usr/lib64/python3.8/copy.py\", line 146, in deepcopy\n[2023-10-26, 17:40:50 UTC] [airflow/utils/log/logging_mixin.py::_propagate_log()::150] WARNING - y = copier(x, memo)\n[2023-10-26, 17:40:50 UTC] [airflow/utils/log/logging_mixin.py::_propagate_log()::150] WARNING - File \"/usr/lib64/python3.8/copy.py\", line 230, in _deepcopy_dict\n[2023-10-26, 17:40:50 UTC] [airflow/utils/log/logging_mixin.py::_propagate_log()::150] WARNING - y[deepcopy(key, memo)] = deepcopy(value, memo)\n[2023-10-26, 17:40:50 UTC] [airflow/utils/log/logging_mixin.py::_propagate_log()::150] WARNING - File \"/usr/lib64/python3.8/copy.py\", line 153, in deepcopy\n[2023-10-26, 17:40:50 UTC] [airflow/utils/log/logging_mixin.py::_propagate_log()::150] WARNING - y = copier(memo)\n[2023-10-26, 17:40:50 UTC] [airflow/utils/log/logging_mixin.py::_propagate_log()::150] WARNING - File \"/home/upgrade/.local/lib/python3.8/site-packages/airflow/models/baseoperator.py\", line 1224, in __deepcopy__\n[2023-10-26, 17:40:50 UTC] [airflow/utils/log/logging_mixin.py::_propagate_log()::150] WARNING - setattr(result, k, copy.deepcopy(v, memo))\n[2023-10-26, 17:40:50 UTC] [airflow/utils/log/logging_mixin.py::_propagate_log()::150] WARNING - File \"/usr/lib64/python3.8/copy.py\", line 146, in deepcopy\n[2023-10-26, 17:40:50 UTC] [airflow/utils/log/logging_mixin.py::_propagate_log()::150] WARNING - y = copier(x, memo)\n[2023-10-26, 17:40:50 UTC] [airflow/utils/log/logging_mixin.py::_propagate_log()::150] WARNING - File \"/usr/lib64/python3.8/copy.py\", line 230, in _deepcopy_dict\n[2023-10-26, 17:40:50 UTC] [airflow/utils/log/logging_mixin.py::_propagate_log()::150] WARNING - y[deepcopy(key, memo)] = deepcopy(value, memo)\n[2023-10-26, 17:40:50 UTC] [airflow/utils/log/logging_mixin.py::_propagate_log()::150] WARNING - File \"/usr/lib64/python3.8/copy.py\", line 161, in deepcopy\n[2023-10-26, 17:40:50 UTC] [airflow/utils/log/logging_mixin.py::_propagate_log()::150] WARNING - rv = reductor(4)\n[2023-10-26, 17:40:50 UTC] [airflow/utils/log/logging_mixin.py::_propagate_log()::150] WARNING - TypeError: cannot pickle 'module' object" + } + ], + "border": 0 + }, + { + "type": "rich_text_section", + "elements": [ + { + "type": "text", + "text": "but with 1.4.1 its stopped processing any further and threw error" + } + ] + } + ] + } + ], + "team": "T01CWUYP5AR", + "thread_ts": "1698340358.557159", + "parent_user_id": "U062WLFMRTP" + }, + { + "type": "message", + "text": "I see the difference of calling in these 2 versions, current versions checks if Airflow is >2.6 then directly runs on_running but earlier version was running on separate thread. IS this what's raising this exception?", + "files": [ + { + "id": "F06300USGUS", + "created": 1698344212, + "timestamp": 1698344212, + "name": "Screenshot 2023-10-26 at 11.16.47 AM.png", + "title": "Screenshot 2023-10-26 at 11.16.47 AM.png", + "mimetype": "image/png", + "filetype": "png", + "pretty_type": "PNG", + "user": "U062WLFMRTP", + "user_team": "T01CWUYP5AR", + "editable": false, + "size": 67732, + "mode": "hosted", + "is_external": false, + "external_type": "", + "is_public": true, + "public_url_shared": false, + "display_as_bot": false, + "username": "", + "url_private": "https://files.slack.com/files-pri/T01CWUYP5AR-F06300USGUS/screenshot_2023-10-26_at_11.16.47_am.png", + "url_private_download": "https://files.slack.com/files-pri/T01CWUYP5AR-F06300USGUS/download/screenshot_2023-10-26_at_11.16.47_am.png", + "media_display_type": "unknown", + "thumb_64": "https://files.slack.com/files-tmb/T01CWUYP5AR-F06300USGUS-7f787b1bdb/screenshot_2023-10-26_at_11.16.47_am_64.png", + "thumb_80": "https://files.slack.com/files-tmb/T01CWUYP5AR-F06300USGUS-7f787b1bdb/screenshot_2023-10-26_at_11.16.47_am_80.png", + "thumb_360": "https://files.slack.com/files-tmb/T01CWUYP5AR-F06300USGUS-7f787b1bdb/screenshot_2023-10-26_at_11.16.47_am_360.png", + "thumb_360_w": 360, + "thumb_360_h": 73, + "thumb_480": "https://files.slack.com/files-tmb/T01CWUYP5AR-F06300USGUS-7f787b1bdb/screenshot_2023-10-26_at_11.16.47_am_480.png", + "thumb_480_w": 480, + "thumb_480_h": 97, + "thumb_160": "https://files.slack.com/files-tmb/T01CWUYP5AR-F06300USGUS-7f787b1bdb/screenshot_2023-10-26_at_11.16.47_am_160.png", + "thumb_720": "https://files.slack.com/files-tmb/T01CWUYP5AR-F06300USGUS-7f787b1bdb/screenshot_2023-10-26_at_11.16.47_am_720.png", + "thumb_720_w": 720, + "thumb_720_h": 145, + "thumb_800": "https://files.slack.com/files-tmb/T01CWUYP5AR-F06300USGUS-7f787b1bdb/screenshot_2023-10-26_at_11.16.47_am_800.png", + "thumb_800_w": 800, + "thumb_800_h": 162, + "thumb_960": "https://files.slack.com/files-tmb/T01CWUYP5AR-F06300USGUS-7f787b1bdb/screenshot_2023-10-26_at_11.16.47_am_960.png", + "thumb_960_w": 960, + "thumb_960_h": 194, + "thumb_1024": "https://files.slack.com/files-tmb/T01CWUYP5AR-F06300USGUS-7f787b1bdb/screenshot_2023-10-26_at_11.16.47_am_1024.png", + "thumb_1024_w": 1024, + "thumb_1024_h": 207, + "original_w": 1723, + "original_h": 348, + "thumb_tiny": "AwAJADDROKXmiloATmloooAQ80c0UtAH/9k=", + "permalink": "https://openlineage.slack.com/files/U062WLFMRTP/F06300USGUS/screenshot_2023-10-26_at_11.16.47_am.png", + "permalink_public": "https://slack-files.com/T01CWUYP5AR-F06300USGUS-3d699164d7", + "is_starred": false, + "has_rich_preview": false, + "file_access": "visible" + } + ], + "upload": false, + "user": "U062WLFMRTP", + "display_as_bot": false, + "ts": "1698344288.616259", + "blocks": [ + { + "type": "rich_text", + "block_id": "DiTSY", + "elements": [ + { + "type": "rich_text_section", + "elements": [ + { + "type": "text", + "text": "I see the difference of calling in these 2 versions, current versions checks if Airflow is >2.6 then directly runs on_running but earlier version was running on separate thread. IS this what's raising this exception?" + } + ] + } + ] + } + ], + "client_msg_id": "0feee1c7-ae91-4997-8a3e-31192460d416", + "thread_ts": "1698340358.557159", + "parent_user_id": "U062WLFMRTP" + }, + { + "client_msg_id": "f3ed8118-02ef-4c05-ab0f-1fb9839c7f48", + "type": "message", + "text": "this is the issue - while copying the task", + "user": "U062WLFMRTP", + "ts": "1698344689.617209", + "blocks": [ + { + "type": "rich_text", + "block_id": "54jfB", + "elements": [ + { + "type": "rich_text_section", + "elements": [ + { + "type": "text", + "text": "this is the issue - " + }, + { + "type": "link", + "url": "https://github.com/OpenLineage/OpenLineage/blob/c343835c1664eda94d5c315897ae6702854c81bd/integration/airflow/openlineage/airflow/listener.py#L89" + }, + { + "type": "text", + "text": " while copying the task" + } + ] + } + ] + } + ], + "team": "T01CWUYP5AR", + "attachments": [ + { + "id": 1, + "footer_icon": "https://slack.github.com/static/img/favicon-neutral.png", + "color": "24292f", + "bot_id": "B01VA0FB340", + "app_unfurl_url": "https://github.com/OpenLineage/OpenLineage/blob/c343835c1664eda94d5c315897ae6702854c81bd/integration/airflow/openlineage/airflow/listener.py#L89", + "is_app_unfurl": true, + "app_id": "A01BP7R4KNY", + "fallback": "", + "text": "```\n task_instance_copy = copy.deepcopy(task_instance)\n```", + "title": "", + "footer": "", + "mrkdwn_in": [ + "text" + ] + } + ], + "thread_ts": "1698340358.557159", + "parent_user_id": "U062WLFMRTP" + }, + { + "client_msg_id": "6359f22f-dc9a-4134-8ea4-6326b48ad13f", + "type": "message", + "text": "since we are directly running if version>2.6.0 therefore its throwing error in main processing", + "user": "U062WLFMRTP", + "ts": "1698344721.996409", + "blocks": [ + { + "type": "rich_text", + "block_id": "AMfCW", + "elements": [ + { + "type": "rich_text_section", + "elements": [ + { + "type": "text", + "text": "since we are directly running if version>2.6.0 therefore its throwing error in main processing" + } + ] + } + ] + } + ], + "team": "T01CWUYP5AR", + "thread_ts": "1698340358.557159", + "parent_user_id": "U062WLFMRTP" + }, + { + "client_msg_id": "f32e38cd-277c-4426-a62e-44947b7fced4", + "type": "message", + "text": "may i know which Airflow version we tested this process?", + "user": "U062WLFMRTP", + "ts": "1698344882.925899", + "blocks": [ + { + "type": "rich_text", + "block_id": "9Vhnq", + "elements": [ + { + "type": "rich_text_section", + "elements": [ + { + "type": "text", + "text": "may i know which Airflow version we tested this process?" + } + ] + } + ] + } + ], + "team": "T01CWUYP5AR", + "thread_ts": "1698340358.557159", + "parent_user_id": "U062WLFMRTP" + }, + { + "client_msg_id": "c29bf7d1-f14f-4a11-be4f-4c34d72b41fc", + "type": "message", + "text": "im on 2.6.3", + "user": "U062WLFMRTP", + "ts": "1698344919.858279", + "blocks": [ + { + "type": "rich_text", + "block_id": "Nq6Px", + "elements": [ + { + "type": "rich_text_section", + "elements": [ + { + "type": "text", + "text": "im on 2.6.3" + } + ] + } + ] + } + ], + "team": "T01CWUYP5AR", + "thread_ts": "1698340358.557159", + "parent_user_id": "U062WLFMRTP" + }, + { + "client_msg_id": "11c94018-1a6d-4559-8cad-d2c306a867c7", + "type": "message", + "text": "2.1.4, 2.2.4, 2.3.4, 2.4.3, 2.5.2, 2.6.1\nusually there are not too many changes between minor versions\n\nI still believe it might be some code you might improve and probably is also an antipattern in airflow", + "user": "U02S6F54MAB", + "ts": "1698345053.182559", + "blocks": [ + { + "type": "rich_text", + "block_id": "tzdJj", + "elements": [ + { + "type": "rich_text_section", + "elements": [ + { + "type": "text", + "text": "2.1.4, 2.2.4, 2.3.4, 2.4.3, 2.5.2, 2.6.1\nusually there are not too many changes between minor versions\n\nI still believe it might be some code you might improve and probably is also an antipattern in airflow" + } + ] + } + ] + } + ], + "team": "T01CWUYP5AR", + "edited": { + "user": "U02S6F54MAB", + "ts": "1698345063.000000" + }, + "thread_ts": "1698340358.557159", + "parent_user_id": "U062WLFMRTP" + }, + { + "client_msg_id": "2c209341-29cc-46c8-a90c-74421a74355e", + "type": "message", + "text": "hummm...that's a valid observation but I dont write DAGS, other teams do, so imagine if many people wrote such DAGS I can't ask everyone to change their patterns right? If something is running on current openlineage version with warning that should still be running on upgraded version isn't it?", + "user": "U062WLFMRTP", + "ts": "1698345266.075579", + "blocks": [ + { + "type": "rich_text", + "block_id": "+DHu0", + "elements": [ + { + "type": "rich_text_section", + "elements": [ + { + "type": "text", + "text": "hummm...that's a valid observation but I dont write DAGS, other teams do, so imagine if many people wrote such DAGS I can't ask everyone to change their patterns right? If something is running on current openlineage version with warning that should still be running on upgraded version isn't it?" + } + ] + } + ] + } + ], + "team": "T01CWUYP5AR", + "edited": { + "user": "U062WLFMRTP", + "ts": "1698345284.000000" + }, + "thread_ts": "1698340358.557159", + "parent_user_id": "U062WLFMRTP" + }, + { + "client_msg_id": "68f6ad78-00fd-4d6a-a3e6-6ee5d8d82504", + "type": "message", + "text": "however I see ur point", + "user": "U062WLFMRTP", + "ts": "1698345484.622449", + "blocks": [ + { + "type": "rich_text", + "block_id": "MQy78", + "elements": [ + { + "type": "rich_text_section", + "elements": [ + { + "type": "text", + "text": "however I see ur point" + } + ] + } + ] + } + ], + "team": "T01CWUYP5AR", + "thread_ts": "1698340358.557159", + "parent_user_id": "U062WLFMRTP" + }, + { + "client_msg_id": "50fe8d3a-8dac-4ca4-a97d-0044a76cef3e", + "type": "message", + "text": "So that specific task has 570 line of query and pretty bulky query, let me split into smaller units", + "user": "U062WLFMRTP", + "ts": "1698346192.384519", + "blocks": [ + { + "type": "rich_text", + "block_id": "z9PWm", + "elements": [ + { + "type": "rich_text_section", + "elements": [ + { + "type": "text", + "text": "So that specific task has 570 line of query and pretty bulky query, let me split into smaller units" + } + ] + } + ] + } + ], + "team": "T01CWUYP5AR", + "thread_ts": "1698340358.557159", + "parent_user_id": "U062WLFMRTP" + }, + { + "client_msg_id": "db15173c-b598-4942-a06a-4eeab1e2de60", + "type": "message", + "text": "that should help right? <@U02S6F54MAB>", + "user": "U062WLFMRTP", + "ts": "1698346215.462899", + "blocks": [ + { + "type": "rich_text", + "block_id": "iXCWL", + "elements": [ + { + "type": "rich_text_section", + "elements": [ + { + "type": "text", + "text": "that should help right? " + }, + { + "type": "user", + "user_id": "U02S6F54MAB" + } + ] + } + ] + } + ], + "team": "T01CWUYP5AR", + "thread_ts": "1698340358.557159", + "parent_user_id": "U062WLFMRTP" + }, + { + "client_msg_id": "ea4541ab-3a19-40d6-8574-6770444fd9dd", + "type": "message", + "text": "query length shouldn’t be the issue, rather any python code", + "user": "U02S6F54MAB", + "ts": "1698346287.209199", + "blocks": [ + { + "type": "rich_text", + "block_id": "09bOT", + "elements": [ + { + "type": "rich_text_section", + "elements": [ + { + "type": "text", + "text": "query length shouldn’t be the issue, rather any python code" + } + ] + } + ] + } + ], + "team": "T01CWUYP5AR", + "thread_ts": "1698340358.557159", + "parent_user_id": "U062WLFMRTP" + }, + { + "client_msg_id": "fb298080-70a7-4d96-a3d6-7550004ee9e9", + "type": "message", + "text": "I get your point too, we might figure out some mechanism to skip irrelevant parts of task instance so that it doesn’t fail then", + "user": "U02S6F54MAB", + "ts": "1698346310.544309", + "blocks": [ + { + "type": "rich_text", + "block_id": "alsM7", + "elements": [ + { + "type": "rich_text_section", + "elements": [ + { + "type": "text", + "text": "I get your point too, we might figure out some mechanism to skip irrelevant parts of task instance so that it doesn’t fail then" + } + ] + } + ] + } + ], + "team": "T01CWUYP5AR", + "thread_ts": "1698340358.557159", + "parent_user_id": "U062WLFMRTP" + }, + { + "client_msg_id": "11174c4d-7251-4007-9757-bccd95258aa1", + "type": "message", + "text": "actually its failing on that task itself", + "user": "U062WLFMRTP", + "ts": "1698346332.553519", + "blocks": [ + { + "type": "rich_text", + "block_id": "RkGPE", + "elements": [ + { + "type": "rich_text_section", + "elements": [ + { + "type": "text", + "text": "actually its failing on that task itself" + } + ] + } + ] + } + ], + "team": "T01CWUYP5AR", + "thread_ts": "1698340358.557159", + "parent_user_id": "U062WLFMRTP" + }, + { + "client_msg_id": "1a2bbd0d-3879-4bac-88d7-b558dbd81482", + "type": "message", + "text": "let me try it will be pretty quick", + "user": "U062WLFMRTP", + "ts": "1698346353.120079", + "blocks": [ + { + "type": "rich_text", + "block_id": "BzN/H", + "elements": [ + { + "type": "rich_text_section", + "elements": [ + { + "type": "text", + "text": "let me try it will be pretty quick" + } + ] + } + ] + } + ], + "team": "T01CWUYP5AR", + "thread_ts": "1698340358.557159", + "parent_user_id": "U062WLFMRTP" + }, + { + "client_msg_id": "083fd308-34be-4b79-9b01-c0a615c191d8", + "type": "message", + "text": "<@U02S6F54MAB> but ur right we have to fix this at Openlineage side as well. Because ideally Openlineage shouldn't be causing any issue to the main DAG processing", + "user": "U062WLFMRTP", + "ts": "1698346738.780429", + "blocks": [ + { + "type": "rich_text", + "block_id": "1G5HT", + "elements": [ + { + "type": "rich_text_section", + "elements": [ + { + "type": "user", + "user_id": "U02S6F54MAB" + }, + { + "type": "text", + "text": " but ur right we have to fix this at Openlineage side as well. Because ideally Openlineage shouldn't be causing any issue to the main DAG processing" + } + ] + } + ] + } + ], + "team": "T01CWUYP5AR", + "thread_ts": "1698340358.557159", + "parent_user_id": "U062WLFMRTP" + }, + { + "client_msg_id": "697ec9cc-e9ba-49d8-9e50-7176369827e4", + "type": "message", + "text": "it doesn’t break any airflow functionality, execution is wrapped into try/except block, only exception traceback is logged as you can see", + "user": "U02S6F54MAB", + "ts": "1698357065.052169", + "blocks": [ + { + "type": "rich_text", + "block_id": "XMsF6", + "elements": [ + { + "type": "rich_text_section", + "elements": [ + { + "type": "text", + "text": "it doesn’t break any airflow functionality, execution is wrapped into try/except block, only exception traceback is logged as you can see" + } + ] + } + ] + } + ], + "team": "T01CWUYP5AR", + "thread_ts": "1698340358.557159", + "parent_user_id": "U062WLFMRTP" + }, + { + "client_msg_id": "618b0415-da4a-41f9-b614-eba5419054b8", + "type": "message", + "text": "Can you migrate to Airflow 2.7 and use `apache-airflow-providers-openlineage`? Ideally we wouldn't make meaningful changes to `openlineage-airflow`", + "user": "U01RA9B5GG2", + "ts": "1698398754.823079", + "blocks": [ + { + "type": "rich_text", + "block_id": "7U7pX", + "elements": [ + { + "type": "rich_text_section", + "elements": [ + { + "type": "text", + "text": "Can you migrate to Airflow 2.7 and use " + }, + { + "type": "text", + "text": "apache-airflow-providers-openlineage", + "style": { + "code": true + } + }, + { + "type": "text", + "text": "? Ideally we wouldn't make meaningful changes to " + }, + { + "type": "text", + "text": "openlineage-airflow", + "style": { + "code": true + } + } + ] + } + ] + } + ], + "team": "T01CWUYP5AR", + "thread_ts": "1698340358.557159", + "parent_user_id": "U062WLFMRTP" + }, + { + "client_msg_id": "c9f5545c-8442-43f7-a73b-0fe2dbdebd35", + "type": "message", + "text": "yup thats what im planning to do", + "user": "U062WLFMRTP", + "ts": "1698420944.617489", + "blocks": [ + { + "type": "rich_text", + "block_id": "tqNAF", + "elements": [ + { + "type": "rich_text_section", + "elements": [ + { + "type": "text", + "text": "yup thats what im planning to do" + } + ] + } + ] + } + ], + "team": "T01CWUYP5AR", + "thread_ts": "1698340358.557159", + "parent_user_id": "U062WLFMRTP" + }, + { + "type": "message", + "subtype": "thread_broadcast", + "text": "referencing to conversation - what it takes to move to openlineage provider package from openlineage-airflow. Im updating Airflow to 2.7.2 but moving off of openlineage-airflow to provider package Im trying to estimate the amount of work it takes, any thoughts? reading change_logs I dont think its too much of a change but please share your thoughts and if somewhere its drafted please do share that as well", + "user": "U062WLFMRTP", + "ts": "1698429543.349989", + "thread_ts": "1698340358.557159", + "root": { + "type": "message", + "text": "Im upgrading the version from openlineage-airflow==0.24.0 to openlineage-airflow 1.4.1 but im seeing the following error, any help is appreciated", + "files": [ + { + "id": "F062ZFJN2UB", + "created": 1698340299, + "timestamp": 1698340299, + "name": "Screenshot 2023-10-26 at 10.11.34 AM.png", + "title": "Screenshot 2023-10-26 at 10.11.34 AM.png", + "mimetype": "image/png", + "filetype": "png", + "pretty_type": "PNG", + "user": "U062WLFMRTP", + "user_team": "T01CWUYP5AR", + "editable": false, + "size": 356434, + "mode": "hosted", + "is_external": false, + "external_type": "", + "is_public": true, + "public_url_shared": false, + "display_as_bot": false, + "username": "", + "url_private": "https://files.slack.com/files-pri/T01CWUYP5AR-F062ZFJN2UB/screenshot_2023-10-26_at_10.11.34_am.png", + "url_private_download": "https://files.slack.com/files-pri/T01CWUYP5AR-F062ZFJN2UB/download/screenshot_2023-10-26_at_10.11.34_am.png", + "media_display_type": "unknown", + "thumb_64": "https://files.slack.com/files-tmb/T01CWUYP5AR-F062ZFJN2UB-0fca0a48ea/screenshot_2023-10-26_at_10.11.34_am_64.png", + "thumb_80": "https://files.slack.com/files-tmb/T01CWUYP5AR-F062ZFJN2UB-0fca0a48ea/screenshot_2023-10-26_at_10.11.34_am_80.png", + "thumb_360": "https://files.slack.com/files-tmb/T01CWUYP5AR-F062ZFJN2UB-0fca0a48ea/screenshot_2023-10-26_at_10.11.34_am_360.png", + "thumb_360_w": 360, + "thumb_360_h": 222, + "thumb_480": "https://files.slack.com/files-tmb/T01CWUYP5AR-F062ZFJN2UB-0fca0a48ea/screenshot_2023-10-26_at_10.11.34_am_480.png", + "thumb_480_w": 480, + "thumb_480_h": 297, + "thumb_160": "https://files.slack.com/files-tmb/T01CWUYP5AR-F062ZFJN2UB-0fca0a48ea/screenshot_2023-10-26_at_10.11.34_am_160.png", + "thumb_720": "https://files.slack.com/files-tmb/T01CWUYP5AR-F062ZFJN2UB-0fca0a48ea/screenshot_2023-10-26_at_10.11.34_am_720.png", + "thumb_720_w": 720, + "thumb_720_h": 445, + "thumb_800": "https://files.slack.com/files-tmb/T01CWUYP5AR-F062ZFJN2UB-0fca0a48ea/screenshot_2023-10-26_at_10.11.34_am_800.png", + "thumb_800_w": 800, + "thumb_800_h": 494, + "thumb_960": "https://files.slack.com/files-tmb/T01CWUYP5AR-F062ZFJN2UB-0fca0a48ea/screenshot_2023-10-26_at_10.11.34_am_960.png", + "thumb_960_w": 960, + "thumb_960_h": 593, + "thumb_1024": "https://files.slack.com/files-tmb/T01CWUYP5AR-F062ZFJN2UB-0fca0a48ea/screenshot_2023-10-26_at_10.11.34_am_1024.png", + "thumb_1024_w": 1024, + "thumb_1024_h": 633, + "original_w": 1756, + "original_h": 1085, + "thumb_tiny": "AwAdADC8etKKQ9acKADFLRRQAUUUUAMPWnCm96UUAOopM+1GaAFopM0ZoA//2Q==", + "permalink": "https://openlineage.slack.com/files/U062WLFMRTP/F062ZFJN2UB/screenshot_2023-10-26_at_10.11.34_am.png", + "permalink_public": "https://slack-files.com/T01CWUYP5AR-F062ZFJN2UB-c8de1a91b2", + "is_starred": false, + "has_rich_preview": false, + "file_access": "visible" + } + ], + "upload": false, + "user": "U062WLFMRTP", + "display_as_bot": false, + "ts": "1698340358.557159", + "blocks": [ + { + "type": "rich_text", + "block_id": "CRXyh", + "elements": [ + { + "type": "rich_text_section", + "elements": [ + { + "type": "text", + "text": "Im upgrading the version from openlineage-airflow==0.24.0 to openlineage-airflow 1.4.1 but im seeing the following error, any help is appreciated" + } + ] + } + ] + } + ], + "client_msg_id": "925bbcea-663c-4480-8809-2e2b3dd06020", + "thread_ts": "1698340358.557159", + "reply_count": 39, + "reply_users_count": 4, + "latest_reply": "1698786461.272129", + "reply_users": [ + "U062WLFMRTP", + "U02S6F54MAB", + "U01RA9B5GG2", + "U04AZ7992SU" + ], + "is_locked": false, + "subscribed": false + }, + "blocks": [ + { + "type": "rich_text", + "block_id": "bMuXM", + "elements": [ + { + "type": "rich_text_section", + "elements": [ + { + "type": "text", + "text": "referencing to " + }, + { + "type": "link", + "url": "https://openlineage.slack.com/archives/C01CK9T7HKR/p1698398754823079?thread_ts=1698340358.557159&cid=C01CK9T7HKR", + "text": "this" + }, + { + "type": "text", + "text": " conversation - what it takes to move to openlineage provider package from openlineage-airflow. Im updating Airflow to 2.7.2 but moving off of openlineage-airflow to provider package Im trying to estimate the amount of work it takes, any thoughts? reading change_logs I dont think its too much of a change but please share your thoughts and if somewhere its drafted please do share that as well" + } + ] + } + ] + } + ], + "client_msg_id": "0135497a-c1fc-4c07-b449-3c9178cfa2a8" + }, + { + "client_msg_id": "33da5b3d-c122-4b04-b221-702067236066", + "type": "message", + "text": "Generally not much - I would maybe think of a operator coverage. For example, for BigQuery old `openlineage-airflow` supports `BigQueryExecuteQueryOperator`. However, new `apache-airflow-providers-openlineage` supports `BigQueryInsertJobOperator` - because it's intended replacement for `BigQueryExecuteQueryOperator` and Airflow community does not want to accept contributions to deprecated operators.", + "user": "U01RA9B5GG2", + "ts": "1698668470.071199", + "blocks": [ + { + "type": "rich_text", + "block_id": "0fZgM", + "elements": [ + { + "type": "rich_text_section", + "elements": [ + { + "type": "text", + "text": "Generally not much - I would maybe think of a operator coverage. For example, for BigQuery old " + }, + { + "type": "text", + "text": "openlineage-airflow", + "style": { + "code": true + } + }, + { + "type": "text", + "text": " supports " + }, + { + "type": "text", + "text": "BigQueryExecuteQueryOperator", + "style": { + "code": true + } + }, + { + "type": "text", + "text": ". However, new " + }, + { + "type": "text", + "text": "apache-airflow-providers-openlineage", + "style": { + "code": true + } + }, + { + "type": "text", + "text": " supports " + }, + { + "type": "text", + "text": "BigQueryInsertJobOperator", + "style": { + "code": true + } + }, + { + "type": "text", + "text": " - because it's intended replacement for " + }, + { + "type": "text", + "text": "BigQueryExecuteQueryOperator", + "style": { + "code": true + } + }, + { + "type": "text", + "text": " and Airflow community does not want to accept contributions to deprecated operators." + } + ] + } + ] + } + ], + "team": "T01CWUYP5AR", + "thread_ts": "1698340358.557159", + "parent_user_id": "U062WLFMRTP", + "reactions": [ + { + "name": "pray", + "users": [ + "U062WLFMRTP" + ], + "count": 1 + } + ] + }, + { + "type": "message", + "subtype": "thread_broadcast", + "text": "one question if someone is around - when im keeping both `openlineage-airflow` and `apache-airflow-providers-openlineage` in my requirement file, i see the following error -\n``` from openlineage.airflow.extractors import Extractors\nModuleNotFoundError: No module named 'openlineage.airflow'```\nany thoughts?", + "user": "U062WLFMRTP", + "ts": "1698778838.540239", + "thread_ts": "1698340358.557159", + "root": { + "type": "message", + "text": "Im upgrading the version from openlineage-airflow==0.24.0 to openlineage-airflow 1.4.1 but im seeing the following error, any help is appreciated", + "files": [ + { + "id": "F062ZFJN2UB", + "created": 1698340299, + "timestamp": 1698340299, + "name": "Screenshot 2023-10-26 at 10.11.34 AM.png", + "title": "Screenshot 2023-10-26 at 10.11.34 AM.png", + "mimetype": "image/png", + "filetype": "png", + "pretty_type": "PNG", + "user": "U062WLFMRTP", + "user_team": "T01CWUYP5AR", + "editable": false, + "size": 356434, + "mode": "hosted", + "is_external": false, + "external_type": "", + "is_public": true, + "public_url_shared": false, + "display_as_bot": false, + "username": "", + "url_private": "https://files.slack.com/files-pri/T01CWUYP5AR-F062ZFJN2UB/screenshot_2023-10-26_at_10.11.34_am.png", + "url_private_download": "https://files.slack.com/files-pri/T01CWUYP5AR-F062ZFJN2UB/download/screenshot_2023-10-26_at_10.11.34_am.png", + "media_display_type": "unknown", + "thumb_64": "https://files.slack.com/files-tmb/T01CWUYP5AR-F062ZFJN2UB-0fca0a48ea/screenshot_2023-10-26_at_10.11.34_am_64.png", + "thumb_80": "https://files.slack.com/files-tmb/T01CWUYP5AR-F062ZFJN2UB-0fca0a48ea/screenshot_2023-10-26_at_10.11.34_am_80.png", + "thumb_360": "https://files.slack.com/files-tmb/T01CWUYP5AR-F062ZFJN2UB-0fca0a48ea/screenshot_2023-10-26_at_10.11.34_am_360.png", + "thumb_360_w": 360, + "thumb_360_h": 222, + "thumb_480": "https://files.slack.com/files-tmb/T01CWUYP5AR-F062ZFJN2UB-0fca0a48ea/screenshot_2023-10-26_at_10.11.34_am_480.png", + "thumb_480_w": 480, + "thumb_480_h": 297, + "thumb_160": "https://files.slack.com/files-tmb/T01CWUYP5AR-F062ZFJN2UB-0fca0a48ea/screenshot_2023-10-26_at_10.11.34_am_160.png", + "thumb_720": "https://files.slack.com/files-tmb/T01CWUYP5AR-F062ZFJN2UB-0fca0a48ea/screenshot_2023-10-26_at_10.11.34_am_720.png", + "thumb_720_w": 720, + "thumb_720_h": 445, + "thumb_800": "https://files.slack.com/files-tmb/T01CWUYP5AR-F062ZFJN2UB-0fca0a48ea/screenshot_2023-10-26_at_10.11.34_am_800.png", + "thumb_800_w": 800, + "thumb_800_h": 494, + "thumb_960": "https://files.slack.com/files-tmb/T01CWUYP5AR-F062ZFJN2UB-0fca0a48ea/screenshot_2023-10-26_at_10.11.34_am_960.png", + "thumb_960_w": 960, + "thumb_960_h": 593, + "thumb_1024": "https://files.slack.com/files-tmb/T01CWUYP5AR-F062ZFJN2UB-0fca0a48ea/screenshot_2023-10-26_at_10.11.34_am_1024.png", + "thumb_1024_w": 1024, + "thumb_1024_h": 633, + "original_w": 1756, + "original_h": 1085, + "thumb_tiny": "AwAdADC8etKKQ9acKADFLRRQAUUUUAMPWnCm96UUAOopM+1GaAFopM0ZoA//2Q==", + "permalink": "https://openlineage.slack.com/files/U062WLFMRTP/F062ZFJN2UB/screenshot_2023-10-26_at_10.11.34_am.png", + "permalink_public": "https://slack-files.com/T01CWUYP5AR-F062ZFJN2UB-c8de1a91b2", + "is_starred": false, + "has_rich_preview": false, + "file_access": "visible" + } + ], + "upload": false, + "user": "U062WLFMRTP", + "display_as_bot": false, + "ts": "1698340358.557159", + "blocks": [ + { + "type": "rich_text", + "block_id": "CRXyh", + "elements": [ + { + "type": "rich_text_section", + "elements": [ + { + "type": "text", + "text": "Im upgrading the version from openlineage-airflow==0.24.0 to openlineage-airflow 1.4.1 but im seeing the following error, any help is appreciated" + } + ] + } + ] + } + ], + "client_msg_id": "925bbcea-663c-4480-8809-2e2b3dd06020", + "thread_ts": "1698340358.557159", + "reply_count": 39, + "reply_users_count": 4, + "latest_reply": "1698786461.272129", + "reply_users": [ + "U062WLFMRTP", + "U02S6F54MAB", + "U01RA9B5GG2", + "U04AZ7992SU" + ], + "is_locked": false, + "subscribed": false + }, + "blocks": [ + { + "type": "rich_text", + "block_id": "7zvcN", + "elements": [ + { + "type": "rich_text_section", + "elements": [ + { + "type": "text", + "text": "one question if someone is around - when im keeping both " + }, + { + "type": "text", + "text": "openlineage-airflow", + "style": { + "code": true + } + }, + { + "type": "text", + "text": " and " + }, + { + "type": "text", + "text": "apache-airflow-providers-openlineage", + "style": { + "code": true + } + }, + { + "type": "text", + "text": " in my requirement file, i see the following error -\n" + } + ] + }, + { + "type": "rich_text_preformatted", + "elements": [ + { + "type": "text", + "text": " from openlineage.airflow.extractors import Extractors\nModuleNotFoundError: No module named 'openlineage.airflow'" + } + ], + "border": 0 + }, + { + "type": "rich_text_section", + "elements": [ + { + "type": "text", + "text": "any thoughts?" + } + ] + } + ] + } + ], + "client_msg_id": "84068da9-05c1-43a6-97a3-cb24d71c5832" + }, + { + "client_msg_id": "5b3abc4b-da1b-4f45-953a-ce8f48a8e43d", + "type": "message", + "text": "I would usually do a `pip freeze | grep openlineage` as a sanity check to validate that the module is actually installed. Not sure how the provider and the module play together though", + "user": "U04AZ7992SU", + "ts": "1698781027.993429", + "blocks": [ + { + "type": "rich_text", + "block_id": "GlF5T", + "elements": [ + { + "type": "rich_text_section", + "elements": [ + { + "type": "text", + "text": "I would usually do a " + }, + { + "type": "text", + "text": "pip freeze | grep openlineage", + "style": { + "code": true + } + }, + { + "type": "text", + "text": " as a sanity check to validate that the module is actually installed. Not sure how the provider and the module play together though" + } + ] + } + ] + } + ], + "team": "T01CWUYP5AR", + "thread_ts": "1698340358.557159", + "parent_user_id": "U062WLFMRTP" + }, + { + "client_msg_id": "ae9c33b6-583e-464c-afa2-257262faa903", + "type": "message", + "text": "yeah so <@U04AZ7992SU> im not getting how i can use the specific extractor when i run my operator. Say for example, I have custom datawarehouseOperator and i want to override get_openlineage_facets_on_start and get_openlineage_facets_on_complete using the redshift extractor then how would i do that?", + "user": "U062WLFMRTP", + "ts": "1698786461.272129", + "blocks": [ + { + "type": "rich_text", + "block_id": "SRKCJ", + "elements": [ + { + "type": "rich_text_section", + "elements": [ + { + "type": "text", + "text": "yeah so " + }, + { + "type": "user", + "user_id": "U04AZ7992SU" + }, + { + "type": "text", + "text": " im not getting how i can use the specific extractor when i run my operator. Say for example, I have custom datawarehouseOperator and i want to override get_openlineage_facets_on_start and get_openlineage_facets_on_complete using the redshift extractor then how would i do that?" + } + ] + } + ] + } + ], + "team": "T01CWUYP5AR", + "thread_ts": "1698340358.557159", + "parent_user_id": "U062WLFMRTP" + } + ] + }, + { + "client_msg_id": "b8ca2008-c764-472b-9c2e-7317379aed21", + "type": "message", + "text": "Hello Team", + "user": "U062WLFMRTP", + "ts": "1698340277.847709", + "blocks": [ + { + "type": "rich_text", + "block_id": "1qV9L", + "elements": [ + { + "type": "rich_text_section", + "elements": [ + { + "type": "text", + "text": "Hello Team" + } + ] + } + ] + } + ], + "team": "T01CWUYP5AR" + }, + { + "client_msg_id": "e233e776-2f62-49df-9b40-c8898974d6ef", + "type": "message", + "text": "Hi I want to customise the events which comes from Openlineage spark . Can some one give some information", + "user": "U062Q95A1FG", + "ts": "1698315220.142929", + "blocks": [ + { + "type": "rich_text", + "block_id": "V6ApU", + "elements": [ + { + "type": "rich_text_section", + "elements": [ + { + "type": "text", + "text": "Hi I want to customise the events which comes from Openlineage spark . Can some one give some information" + } + ] + } + ] + } + ], + "team": "T01CWUYP5AR", + "thread_ts": "1698315220.142929", + "reply_count": 2, + "reply_users_count": 2, + "latest_reply": "1698328387.014389", + "reply_users": [ + "U02MK6YNAQ5", + "U062Q95A1FG" + ], + "is_locked": false, + "subscribed": false, + "replies": [ + { + "client_msg_id": "d2353d60-c8ea-43cb-953a-b3287837757e", + "type": "message", + "text": "Hi <@U062Q95A1FG>, please get familiar with `Extending` section on our docs: ", + "user": "U02MK6YNAQ5", + "ts": "1698320741.327129", + "blocks": [ + { + "type": "rich_text", + "block_id": "pwnrB", + "elements": [ + { + "type": "rich_text_section", + "elements": [ + { + "type": "text", + "text": "Hi " + }, + { + "type": "user", + "user_id": "U062Q95A1FG" + }, + { + "type": "text", + "text": ", please get familiar with " + }, + { + "type": "text", + "text": "Extending", + "style": { + "code": true + } + }, + { + "type": "text", + "text": " section on our docs: " + }, + { + "type": "link", + "url": "https://github.com/OpenLineage/OpenLineage/tree/main/integration/spark#extending" + } + ] + } + ] + } + ], + "team": "T01CWUYP5AR", + "thread_ts": "1698315220.142929", + "parent_user_id": "U062Q95A1FG" + }, + { + "client_msg_id": "1721245b-dcc9-4d3c-8fba-31f7c672dc6c", + "type": "message", + "text": "Okay thank you. Just checking any other docs or git code which also can help me", + "user": "U062Q95A1FG", + "ts": "1698328387.014389", + "blocks": [ + { + "type": "rich_text", + "block_id": "jvKNE", + "elements": [ + { + "type": "rich_text_section", + "elements": [ + { + "type": "text", + "text": "Okay thank you. Just checking any other docs or git code which also can help me" + } + ] + } + ] + } + ], + "team": "T01CWUYP5AR", + "thread_ts": "1698315220.142929", + "parent_user_id": "U062Q95A1FG" + } + ] + }, + { + "client_msg_id": "45451dcf-7e95-4986-92f0-5bda253f541b", + "type": "message", + "text": "Hi,\n\nWe are using openlineage spark connector. We have used spark 3.2 and scala 2.12 so far. We have triggered a new job with Spark 3.4 and scala 2.13 and faced below exception.\n\n\n```java.lang.NoSuchMethodError: 'scala.collection.Seq org.apache.spark.sql.catalyst.plans.logical.LogicalPlan.map(scala.Function1)'\n\tat io.openlineage.spark.agent.lifecycle.OpenLineageRunEventBuilder.lambda$buildInputDatasets$6(OpenLineageRunEventBuilder.java:341)\n\tat java.base/java.util.Optional.map(Optional.java:265)\n\tat io.openlineage.spark.agent.lifecycle.OpenLineageRunEventBuilder.buildInputDatasets(OpenLineageRunEventBuilder.java:339)\n\tat io.openlineage.spark.agent.lifecycle.OpenLineageRunEventBuilder.populateRun(OpenLineageRunEventBuilder.java:295)\n\tat io.openlineage.spark.agent.lifecycle.OpenLineageRunEventBuilder.buildRun(OpenLineageRunEventBuilder.java:279)\n\tat io.openlineage.spark.agent.lifecycle.OpenLineageRunEventBuilder.buildRun(OpenLineageRunEventBuilder.java:222)\n\tat io.openlineage.spark.agent.lifecycle.SparkSQLExecutionContext.start(SparkSQLExecutionContext.java:72)\n\tat io.openlineage.spark.agent.OpenLineageSparkListener.lambda$sparkSQLExecStart$0(OpenLineageSparkListener.java:91)```", + "user": "U0625RZ7KR9", + "ts": "1697840317.080859", + "blocks": [ + { + "type": "rich_text", + "block_id": "mnVGE", + "elements": [ + { + "type": "rich_text_section", + "elements": [ + { + "type": "text", + "text": "Hi,\n\nWe are using openlineage spark connector. We have used spark 3.2 and scala 2.12 so far. We have triggered a new job with Spark 3.4 and scala 2.13 and faced below exception.\n\n\n" + } + ] + }, + { + "type": "rich_text_preformatted", + "elements": [ + { + "type": "text", + "text": "java.lang.NoSuchMethodError: 'scala.collection.Seq org.apache.spark.sql.catalyst.plans.logical.LogicalPlan.map(scala.Function1)'\n\tat io.openlineage.spark.agent.lifecycle.OpenLineageRunEventBuilder.lambda$buildInputDatasets$6(OpenLineageRunEventBuilder.java:341)\n\tat java.base/java.util.Optional.map(Optional.java:265)\n\tat io.openlineage.spark.agent.lifecycle.OpenLineageRunEventBuilder.buildInputDatasets(OpenLineageRunEventBuilder.java:339)\n\tat io.openlineage.spark.agent.lifecycle.OpenLineageRunEventBuilder.populateRun(OpenLineageRunEventBuilder.java:295)\n\tat io.openlineage.spark.agent.lifecycle.OpenLineageRunEventBuilder.buildRun(OpenLineageRunEventBuilder.java:279)\n\tat io.openlineage.spark.agent.lifecycle.OpenLineageRunEventBuilder.buildRun(OpenLineageRunEventBuilder.java:222)\n\tat io.openlineage.spark.agent.lifecycle.SparkSQLExecutionContext.start(SparkSQLExecutionContext.java:72)\n\tat io.openlineage.spark.agent.OpenLineageSparkListener.lambda$sparkSQLExecStart$0(OpenLineageSparkListener.java:91)" + } + ], + "border": 0 + } + ] + } + ], + "team": "T01CWUYP5AR", + "thread_ts": "1697840317.080859", + "reply_count": 3, + "reply_users_count": 2, + "latest_reply": "1698144434.527919", + "reply_users": [ + "U02MK6YNAQ5", + "U0625RZ7KR9" + ], + "is_locked": false, + "subscribed": false, + "replies": [ + { + "client_msg_id": "42e1e997-a71f-4c3a-8692-ef6efc8b53e2", + "type": "message", + "text": "Hmy, that is interesting. Did it occur on databricks runtime? Could you give it a try with Scala 2.12? I think we don't test scala 2.13.", + "user": "U02MK6YNAQ5", + "ts": "1698051385.420489", + "blocks": [ + { + "type": "rich_text", + "block_id": "ydjNi", + "elements": [ + { + "type": "rich_text_section", + "elements": [ + { + "type": "text", + "text": "Hmy, that is interesting. Did it occur on databricks runtime? Could you give it a try with Scala 2.12? I think we don't test scala 2.13." + } + ] + } + ] + } + ], + "team": "T01CWUYP5AR", + "thread_ts": "1697840317.080859", + "parent_user_id": "U0625RZ7KR9" + }, + { + "client_msg_id": "2bfd49e9-104f-473b-8264-635a9f8d7581", + "type": "message", + "text": "I believe our Scala 2.12 jobs are working fine. It's not databricks runtime. We run Spark on Kube.", + "user": "U0625RZ7KR9", + "ts": "1698076933.905129", + "blocks": [ + { + "type": "rich_text", + "block_id": "GvRot", + "elements": [ + { + "type": "rich_text_section", + "elements": [ + { + "type": "text", + "text": "I believe our Scala 2.12 jobs are working fine. It's not databricks runtime. We run Spark on Kube." + } + ] + } + ] + } + ], + "team": "T01CWUYP5AR", + "thread_ts": "1697840317.080859", + "parent_user_id": "U0625RZ7KR9" + }, + { + "client_msg_id": "81acde74-8d20-41f1-8d8e-175ab917bc11", + "type": "message", + "text": "Ok. I think You can raise an issue to support Scala 2.13 for latest Spark versions.", + "user": "U02MK6YNAQ5", + "ts": "1698144434.527919", + "blocks": [ + { + "type": "rich_text", + "block_id": "cx3Rw", + "elements": [ + { + "type": "rich_text_section", + "elements": [ + { + "type": "text", + "text": "Ok. I think You can raise an issue to support Scala 2.13 for latest Spark versions." + } + ] + } + ] + } + ], + "team": "T01CWUYP5AR", + "thread_ts": "1697840317.080859", + "parent_user_id": "U0625RZ7KR9" + } + ] + }, + { + "type": "message", + "subtype": "thread_broadcast", + "text": "<@U05JY6MN8MS>\nI am trying to contribute to Integration tests which is listed here as \nthe mentions that i can trigger CI for integration tests from forked branch.\n.\nbut i am unable to do so, is there a way to trigger CI from forked brach or do i have to get permission from someone to run the CI?\n\ni am getting this error when i run this command `sudo git-push-fork-to-upstream-branch upstream savannavalgi:hacktober`\n> ```Username for '': savannavalgi\n> Password for '': \n> remote: Permission to OpenLineage/OpenLineage.git denied to savannavalgi.\n> fatal: unable to access '': The requested URL returned error: 403```\ni have tried to configure ssh key\nalso tried to trigger CI from another brach,\nand tried all of this after fetching the latest upstream\n\ncc: <@U05JBHLPY8K> <@U01RA9B5GG2> <@U05HD9G5T17>", + "user": "U05KCF3EEUR", + "ts": "1697805105.047909", + "thread_ts": "1691660447.094739", + "root": { + "client_msg_id": "14a34e34-0a7d-4cb0-8a7b-611253e00187", + "type": "message", + "text": "Hi,\nAre there any ways to save list of string directly in the dataset facets? Such as the *myfacets* field in this dict\n``` \"facets\": {\n \"metadata_facet\": {\n \"_producer\": \"\",\n \"_schemaURL\": \"\",\n \"myfacets\": [\"a\", \"b\", \"c\"]\n }\n }```", + "user": "U05HD9G5T17", + "ts": "1691660447.094739", + "blocks": [ + { + "type": "rich_text", + "block_id": "aL6sO", + "elements": [ + { + "type": "rich_text_section", + "elements": [ + { + "type": "text", + "text": "Hi,\nAre there any ways to save list of string directly in the dataset facets? Such as the " + }, + { + "type": "text", + "text": "myfacets ", + "style": { + "bold": true + } + }, + { + "type": "text", + "text": "field in this dict\n" + } + ] + }, + { + "type": "rich_text_preformatted", + "elements": [ + { + "type": "text", + "text": " \"facets\": {\n \"metadata_facet\": {\n \"_producer\": \"" + }, + { + "type": "link", + "url": "https://github.com/OpenLineage/OpenLineage/tree/0.29.2/client/python" + }, + { + "type": "text", + "text": "\",\n \"_schemaURL\": \"" + }, + { + "type": "link", + "url": "https://sth/schemas/facets.json#/definitions/SomeFacet" + }, + { + "type": "text", + "text": "\",\n \"myfacets\": [\"a\", \"b\", \"c\"]\n }\n }" + } + ], + "border": 0 + } + ] + } + ], + "team": "T01CWUYP5AR", + "thread_ts": "1691660447.094739", + "reply_count": 29, + "reply_users_count": 4, + "latest_reply": "1700033510.906469", + "reply_users": [ + "U05HD9G5T17", + "U01RA9B5GG2", + "U05KCF3EEUR", + "U02MK6YNAQ5" + ], + "is_locked": false, + "subscribed": false + }, + "blocks": [ + { + "type": "rich_text", + "block_id": "OobjK", + "elements": [ + { + "type": "rich_text_section", + "elements": [ + { + "type": "user", + "user_id": "U05JY6MN8MS" + }, + { + "type": "text", + "text": "\nI am trying to contribute to Integration tests which is listed here as " + }, + { + "type": "link", + "url": "https://github.com/OpenLineage/OpenLineage/issues/2143", + "text": "good first issue" + }, + { + "type": "text", + "text": "\nthe " + }, + { + "type": "link", + "url": "https://github.com/OpenLineage/OpenLineage/blob/main/CONTRIBUTING.md#triggering-ci-runs-from-forks-committers", + "text": "CONTRIBUTING.md ", + "unsafe": true + }, + { + "type": "text", + "text": "mentions that i can trigger CI for integration tests from forked branch.\n" + }, + { + "type": "link", + "url": "https://github.com/jklukas/git-push-fork-to-upstream-branch/blob/master/README.md#git-push-fork-to-upstream-branch", + "text": "using this tool" + }, + { + "type": "text", + "text": ".\nbut i am unable to do so, is there a way to trigger CI from forked brach or do i have to get permission from someone to run the CI?\n\ni am getting this error when i run this command " + }, + { + "type": "text", + "text": "sudo git-push-fork-to-upstream-branch upstream savannavalgi:hacktober", + "style": { + "code": true + } + }, + { + "type": "text", + "text": "\n" + } + ] + }, + { + "type": "rich_text_preformatted", + "elements": [ + { + "type": "text", + "text": "Username for '" + }, + { + "type": "link", + "url": "https://github.com" + }, + { + "type": "text", + "text": "': savannavalgi\nPassword for '" + }, + { + "type": "link", + "url": "https://savannavalgi@github.com" + }, + { + "type": "text", + "text": "': \nremote: Permission to OpenLineage/OpenLineage.git denied to savannavalgi.\nfatal: unable to access '" + }, + { + "type": "link", + "url": "https://github.com/OpenLineage/OpenLineage.git/" + }, + { + "type": "text", + "text": "': The requested URL returned error: 403" + } + ], + "border": 1 + }, + { + "type": "rich_text_section", + "elements": [ + { + "type": "text", + "text": "\ni have tried to configure ssh key\nalso tried to trigger CI from another brach,\nand tried all of this after fetching the latest upstream\n\ncc: " + }, + { + "type": "user", + "user_id": "U05JBHLPY8K" + }, + { + "type": "text", + "text": " " + }, + { + "type": "user", + "user_id": "U01RA9B5GG2" + }, + { + "type": "text", + "text": " " + }, + { + "type": "user", + "user_id": "U05HD9G5T17" + } + ] + } + ] + } + ], + "client_msg_id": "28915632-8aef-451e-8c4a-53e9a9820670" + }, + { + "client_msg_id": "16f5d002-7473-4a9f-9e9c-3a2021e9f62d", + "type": "message", + "text": "Hey all - we've been noticing that some events go unreported by openlineage (spark) when the `AsyncEventQueue` fills up and starts dropping events. Wondering if anyone has experienced this before, and knows why it is happening? We've expanded the event queue capacity and thrown more hardware at the problem but no dice\n\nAlso as a note, the query plans from this job are pretty big - could the listener just be choking up? Happy to open a github issue as well if we suspect that it could be the listener itself having issues", + "user": "U03D8K119LJ", + "ts": "1697742042.953399", + "blocks": [ + { + "type": "rich_text", + "block_id": "ya3F6", + "elements": [ + { + "type": "rich_text_section", + "elements": [ + { + "type": "text", + "text": "Hey all - we've been noticing that some events go unreported by openlineage (spark) when the " + }, + { + "type": "text", + "text": "AsyncEventQueue", + "style": { + "code": true + } + }, + { + "type": "text", + "text": " fills up and starts dropping events. Wondering if anyone has experienced this before, and knows why it is happening? We've expanded the event queue capacity and thrown more hardware at the problem but no dice\n\nAlso as a note, the query plans from this job are pretty big - could the listener just be choking up? Happy to open a github issue as well if we suspect that it could be the listener itself having issues" + } + ] + } + ] + } + ], + "team": "T01CWUYP5AR", + "thread_ts": "1697742042.953399", + "reply_count": 3, + "reply_users_count": 3, + "latest_reply": "1698184226.297699", + "reply_users": [ + "U04EZ2LPDV4", + "U01RA9B5GG2", + "U03D8K119LJ" + ], + "is_locked": false, + "subscribed": false, + "replies": [ + { + "client_msg_id": "9192477b-a1fe-4ecb-bcab-8f9cc3750c42", + "type": "message", + "text": "Hi, just checking, are you excluding the sparkPlan from the events? Or is it sending the spark plan too", + "user": "U04EZ2LPDV4", + "ts": "1697785070.297519", + "blocks": [ + { + "type": "rich_text", + "block_id": "QPMnz", + "elements": [ + { + "type": "rich_text_section", + "elements": [ + { + "type": "text", + "text": "Hi, just checking, are you excluding the sparkPlan from the events? Or is it sending the spark plan too" + } + ] + } + ] + } + ], + "team": "T01CWUYP5AR", + "thread_ts": "1697742042.953399", + "parent_user_id": "U03D8K119LJ" + }, + { + "client_msg_id": "1e190877-fc18-4887-b52d-2c43032062b2", + "type": "message", + "text": "yeah - setting `spark.openlineage.facets.disabled` to `[spark_unknown;spark.logicalPlan]` should help", + "user": "U01RA9B5GG2", + "ts": "1698076780.844459", + "blocks": [ + { + "type": "rich_text", + "block_id": "BnQB8", + "elements": [ + { + "type": "rich_text_section", + "elements": [ + { + "type": "text", + "text": "yeah - setting " + }, + { + "type": "text", + "text": "spark.openlineage.facets.disabled", + "style": { + "code": true + } + }, + { + "type": "text", + "text": " to " + }, + { + "type": "text", + "text": "[spark_unknown;spark.logicalPlan]", + "style": { + "code": true + } + }, + { + "type": "text", + "text": " should help" + } + ] + } + ] + } + ], + "team": "T01CWUYP5AR", + "thread_ts": "1697742042.953399", + "parent_user_id": "U03D8K119LJ" + }, + { + "client_msg_id": "357b5eb1-5099-476e-8572-d2773e8a3130", + "type": "message", + "text": "sorry for the late reply - turns out this job is just whack :smile: we were going in circles trying to figure it out, we end up dropping events without open lineage enabled at all. But good to know that disabling the logical plan should speed us up if we run into this again", + "user": "U03D8K119LJ", + "ts": "1698184226.297699", + "blocks": [ + { + "type": "rich_text", + "block_id": "sjqYh", + "elements": [ + { + "type": "rich_text_section", + "elements": [ + { + "type": "text", + "text": "sorry for the late reply - turns out this job is just whack " + }, + { + "type": "emoji", + "name": "smile", + "unicode": "1f604" + }, + { + "type": "text", + "text": " we were going in circles trying to figure it out, we end up dropping events without open lineage enabled at all. But good to know that disabling the logical plan should speed us up if we run into this again" + } + ] + } + ] + } + ], + "team": "T01CWUYP5AR", + "thread_ts": "1697742042.953399", + "parent_user_id": "U03D8K119LJ" + } + ] + }, + { + "client_msg_id": "d9fbe7bd-c99b-492c-9cf2-85c33e75eced", + "type": "message", + "text": "Hello All, I am completely new for Openlineage, I have to setup the lab to conduct POC on various aspects like Lineage, metadata management , etc. As per openlineage site, i tried downloading Ubuntu, docker and binary files for Marquez. But I am lost somewhere and unable to configure whole setup. Can someone please assist in steps to start from scratch so that i can delve into the Openlineage capabilities. Many thanks", + "user": "U0616K9TSTZ", + "ts": "1697597823.663129", + "blocks": [ + { + "type": "rich_text", + "block_id": "lC/3j", + "elements": [ + { + "type": "rich_text_section", + "elements": [ + { + "type": "text", + "text": "Hello All, I am completely new for Openlineage, I have to setup the lab to conduct POC on various aspects like Lineage, metadata management , etc. As per openlineage site, i tried downloading Ubuntu, docker and binary files for Marquez. But I am lost somewhere and unable to configure whole setup. Can someone please assist in steps to start from scratch so that i can delve into the Openlineage capabilities. Many thanks" + } + ] + } + ] + } + ], + "team": "T01CWUYP5AR", + "thread_ts": "1697597823.663129", + "reply_count": 9, + "reply_users_count": 3, + "latest_reply": "1698149757.690949", + "reply_users": [ + "U02S6F54MAB", + "U02LXF3HUN7", + "U0616K9TSTZ" + ], + "is_locked": false, + "subscribed": true, + "last_read": "1698149757.690949", + "replies": [ + { + "client_msg_id": "6118a046-6700-47a4-bbf7-6096255a0f00", + "type": "message", + "text": "hey, did you try to follow one of these guides?\n", + "user": "U02S6F54MAB", + "ts": "1697607121.185689", + "blocks": [ + { + "type": "rich_text", + "block_id": "YSQ4/", + "elements": [ + { + "type": "rich_text_section", + "elements": [ + { + "type": "text", + "text": "hey, did you try to follow one of these guides?\n" + }, + { + "type": "link", + "url": "https://openlineage.io/docs/guides/about" + } + ] + } + ] + } + ], + "team": "T01CWUYP5AR", + "thread_ts": "1697597823.663129", + "parent_user_id": "U0616K9TSTZ" + }, + { + "client_msg_id": "4aca26b5-9a81-4272-b4c1-2e1256b8277b", + "type": "message", + "text": "Which guide were you using, and what errors/issues are you encountering?", + "user": "U02LXF3HUN7", + "ts": "1697634848.846019", + "blocks": [ + { + "type": "rich_text", + "block_id": "HiX85", + "elements": [ + { + "type": "rich_text_section", + "elements": [ + { + "type": "text", + "text": "Which guide were you using, and what errors/issues are you encountering?" + } + ] + } + ] + } + ], + "team": "T01CWUYP5AR", + "thread_ts": "1697597823.663129", + "parent_user_id": "U0616K9TSTZ" + }, + { + "client_msg_id": "47dc607e-b6d1-42b4-ab76-70c9f1232301", + "type": "message", + "text": "Thanks Jakub for the response.", + "user": "U0616K9TSTZ", + "ts": "1697917394.485199", + "blocks": [ + { + "type": "rich_text", + "block_id": "zwKVv", + "elements": [ + { + "type": "rich_text_section", + "elements": [ + { + "type": "text", + "text": "Thanks Jakub for the response." + } + ] + } + ] + } + ], + "team": "T01CWUYP5AR", + "thread_ts": "1697597823.663129", + "parent_user_id": "U0616K9TSTZ" + }, + { + "type": "message", + "text": "In docker, marquez-api image is not running and exiting with the exit code 127.", + "files": [ + { + "id": "F062L6J1PU1", + "created": 1697917536, + "timestamp": 1697917536, + "name": "image.png", + "title": "image.png", + "mimetype": "image/png", + "filetype": "png", + "pretty_type": "PNG", + "user": "U0616K9TSTZ", + "user_team": "T01CWUYP5AR", + "editable": false, + "size": 166027, + "mode": "hosted", + "is_external": false, + "external_type": "", + "is_public": true, + "public_url_shared": false, + "display_as_bot": false, + "username": "", + "url_private": "https://files.slack.com/files-pri/T01CWUYP5AR-F062L6J1PU1/image.png", + "url_private_download": "https://files.slack.com/files-pri/T01CWUYP5AR-F062L6J1PU1/download/image.png", + "media_display_type": "unknown", + "thumb_64": "https://files.slack.com/files-tmb/T01CWUYP5AR-F062L6J1PU1-5ea4ac9cc0/image_64.png", + "thumb_80": "https://files.slack.com/files-tmb/T01CWUYP5AR-F062L6J1PU1-5ea4ac9cc0/image_80.png", + "thumb_360": "https://files.slack.com/files-tmb/T01CWUYP5AR-F062L6J1PU1-5ea4ac9cc0/image_360.png", + "thumb_360_w": 360, + "thumb_360_h": 141, + "thumb_480": "https://files.slack.com/files-tmb/T01CWUYP5AR-F062L6J1PU1-5ea4ac9cc0/image_480.png", + "thumb_480_w": 480, + "thumb_480_h": 188, + "thumb_160": "https://files.slack.com/files-tmb/T01CWUYP5AR-F062L6J1PU1-5ea4ac9cc0/image_160.png", + "thumb_720": "https://files.slack.com/files-tmb/T01CWUYP5AR-F062L6J1PU1-5ea4ac9cc0/image_720.png", + "thumb_720_w": 720, + "thumb_720_h": 282, + "thumb_800": "https://files.slack.com/files-tmb/T01CWUYP5AR-F062L6J1PU1-5ea4ac9cc0/image_800.png", + "thumb_800_w": 800, + "thumb_800_h": 313, + "thumb_960": "https://files.slack.com/files-tmb/T01CWUYP5AR-F062L6J1PU1-5ea4ac9cc0/image_960.png", + "thumb_960_w": 960, + "thumb_960_h": 375, + "thumb_1024": "https://files.slack.com/files-tmb/T01CWUYP5AR-F062L6J1PU1-5ea4ac9cc0/image_1024.png", + "thumb_1024_w": 1024, + "thumb_1024_h": 400, + "original_w": 1918, + "original_h": 750, + "thumb_tiny": "AwASADC4IIu6CgwQ9Ai5qQUz+P8AGndisgFvD3jWj7PD/wA81p9KvU0XYWQz7PF/zzWmG2XPCpj021YpGGRgHFF2FkJTh0ptOHSkMKKKKACiiigD/9k=", + "permalink": "https://openlineage.slack.com/files/U0616K9TSTZ/F062L6J1PU1/image.png", + "permalink_public": "https://slack-files.com/T01CWUYP5AR-F062L6J1PU1-cba5b6d4fe", + "is_starred": false, + "has_rich_preview": false, + "file_access": "visible" + } + ], + "upload": false, + "user": "U0616K9TSTZ", + "display_as_bot": false, + "ts": "1697917542.984189", + "blocks": [ + { + "type": "rich_text", + "block_id": "BhBOi", + "elements": [ + { + "type": "rich_text_section", + "elements": [ + { + "type": "text", + "text": "In docker, marquez-api image is not running and exiting with the exit code 127." + } + ] + } + ] + } + ], + "client_msg_id": "7a27a78a-6dd9-4185-bed0-b201bfaed0ec", + "thread_ts": "1697597823.663129", + "parent_user_id": "U0616K9TSTZ" + }, + { + "client_msg_id": "67e5bf58-e710-4dd4-becb-423cefa94d87", + "type": "message", + "text": "<@U0616K9TSTZ> thanks. I don't recognize 127, but 9 times out of 10 if the API or DB container fails the reason is a port conflict. Have you checked if port 5000 is available?", + "user": "U02LXF3HUN7", + "ts": "1697981693.157019", + "blocks": [ + { + "type": "rich_text", + "block_id": "QbuKM", + "elements": [ + { + "type": "rich_text_section", + "elements": [ + { + "type": "user", + "user_id": "U0616K9TSTZ" + }, + { + "type": "text", + "text": " thanks. I don't recognize 127, but 9 times out of 10 if the API or DB container fails the reason is a port conflict. Have you checked if port 5000 is available?" + } + ] + } + ] + } + ], + "team": "T01CWUYP5AR", + "thread_ts": "1697597823.663129", + "parent_user_id": "U0616K9TSTZ" + }, + { + "client_msg_id": "34660cf6-f9c6-4401-9a24-bb5655b70cd9", + "type": "message", + "text": "could you please check what’s the output of\n```git config --get core.autocrlf```\nor\n```git config --global --get core.autocrlf```\n?", + "user": "U02S6F54MAB", + "ts": "1697982850.343079", + "blocks": [ + { + "type": "rich_text", + "block_id": "MohY5", + "elements": [ + { + "type": "rich_text_section", + "elements": [ + { + "type": "text", + "text": "could you please check what’s the output of\n" + } + ] + }, + { + "type": "rich_text_preformatted", + "elements": [ + { + "type": "text", + "text": "git config --get core.autocrlf" + } + ], + "border": 0 + }, + { + "type": "rich_text_section", + "elements": [ + { + "type": "text", + "text": "or\n" + } + ] + }, + { + "type": "rich_text_preformatted", + "elements": [ + { + "type": "text", + "text": "git config --global --get core.autocrlf" + } + ], + "border": 0 + }, + { + "type": "rich_text_section", + "elements": [ + { + "type": "text", + "text": "?" + } + ] + } + ] + } + ], + "team": "T01CWUYP5AR", + "thread_ts": "1697597823.663129", + "parent_user_id": "U0616K9TSTZ" + }, + { + "client_msg_id": "410daf0d-94de-420d-8379-2aacdc9d3acd", + "type": "message", + "text": "<@U02LXF3HUN7> thanks , I checked the port 5000 is not available.\nI tried deleting docker images and recreating them, but still the same issue persist stating \n/Usr/bin/env bash/r not found.\nGradle build is successful.", + "user": "U0616K9TSTZ", + "ts": "1698149354.900009", + "blocks": [ + { + "type": "rich_text", + "block_id": "r8GTZ", + "elements": [ + { + "type": "rich_text_section", + "elements": [ + { + "type": "user", + "user_id": "U02LXF3HUN7" + }, + { + "type": "text", + "text": " thanks , I checked the port 5000 is not available.\nI tried deleting docker images and recreating them, but still the same issue persist stating \n/Usr/bin/env bash/r not found.\nGradle build is successful." + } + ] + } + ] + } + ], + "team": "T01CWUYP5AR", + "thread_ts": "1697597823.663129", + "parent_user_id": "U0616K9TSTZ" + }, + { + "client_msg_id": "bccd4e80-b875-49ce-a005-a684ae3d51a2", + "type": "message", + "text": "<@U02S6F54MAB> thanks, first command resulted as true and second command has no response", + "user": "U0616K9TSTZ", + "ts": "1698149394.624269", + "blocks": [ + { + "type": "rich_text", + "block_id": "AqGIf", + "elements": [ + { + "type": "rich_text_section", + "elements": [ + { + "type": "user", + "user_id": "U02S6F54MAB" + }, + { + "type": "text", + "text": " thanks, first command resulted as true and second command has no response" + } + ] + } + ] + } + ], + "team": "T01CWUYP5AR", + "thread_ts": "1697597823.663129", + "parent_user_id": "U0616K9TSTZ" + }, + { + "client_msg_id": "93acc1f4-0c76-4d27-bdba-c504bd769a81", + "type": "message", + "text": "are you running docker and git in Windows or Mac OS before 10.0?", + "user": "U02S6F54MAB", + "ts": "1698149757.690949", + "blocks": [ + { + "type": "rich_text", + "block_id": "0wwJv", + "elements": [ + { + "type": "rich_text_section", + "elements": [ + { + "type": "text", + "text": "are you running docker and git in Windows or Mac OS before 10.0?" + } + ] + } + ] + } + ], + "team": "T01CWUYP5AR", + "edited": { + "user": "U02S6F54MAB", + "ts": "1698150048.000000" + }, + "thread_ts": "1697597823.663129", + "parent_user_id": "U0616K9TSTZ" + } + ] + }, + { + "client_msg_id": "6fd21064-3869-4712-a777-db2a9524a72e", + "type": "message", + "text": "Hello All :wave:!\nWe are currently trying to work the the *spark integration for OpenLineage in our Databricks instance*. The general setup is done and working with a few hicups here and there.\nBut one thing we are still struggling is how to link all spark jobs events with a Databricks job or a notebook run.\nWe´ve recently noticed that some of the events produced by OL have the \"environment-properties\" attribute with information (for our context) regarding notebook path (if it is a notebook run), or the the job run ID (if its a databricks job run). But the thing is that _these attributes are not always present._\nI ran some samples yesterday for a job with 4 notebook tasks. From all 20 json payload sent by the OL listener, only 3 presented the \"environment-properties\" attribute. Its not only happening with Databricks jobs. When i run single notebooks and each cell has its onw set of spark jobs, not all json events presented that property either.\n\nSo my question is *what is the criteria to have this attributes present or not in the event json file*? Or maybe this in an issue? <@U05T8BJD4DU> did you find out anything about this?\n\n:gear: Spark 3.4 / OL-Spark 1.4.1", + "user": "U05TU0U224A", + "ts": "1697527077.180169", + "blocks": [ + { + "type": "rich_text", + "block_id": "BLL64", + "elements": [ + { + "type": "rich_text_section", + "elements": [ + { + "type": "text", + "text": "Hello All " + }, + { + "type": "emoji", + "name": "wave", + "unicode": "1f44b" + }, + { + "type": "text", + "text": "!\nWe are currently trying to work the the " + }, + { + "type": "text", + "text": "spark integration for OpenLineage in our Databricks instance", + "style": { + "bold": true + } + }, + { + "type": "text", + "text": ". The general setup is done and working with a few hicups here and there.\nBut one thing we are still struggling is how to link all spark jobs events with a Databricks job or a notebook run.\nWe´ve recently noticed that some of the events produced by OL have the \"environment-properties\" attribute with information (for our context) regarding notebook path (if it is a notebook run), or the the job run ID (if its a databricks job run). But the thing is that " + }, + { + "type": "text", + "text": "these attributes are not always present.", + "style": { + "italic": true + } + }, + { + "type": "text", + "text": "\nI ran some samples yesterday for a job with 4 notebook tasks. From all 20 json payload sent by the OL listener, only 3 presented the \"environment-properties\" attribute. Its not only happening with Databricks jobs. When i run single notebooks and each cell has its onw set of spark jobs, not all json events presented that property either.\n\nSo my question is " + }, + { + "type": "text", + "text": "what is the criteria to have this attributes present or not in the event json file", + "style": { + "bold": true + } + }, + { + "type": "text", + "text": "? Or maybe this in an issue? " + }, + { + "type": "user", + "user_id": "U05T8BJD4DU" + }, + { + "type": "text", + "text": " did you find out anything about this?\n\n" + }, + { + "type": "emoji", + "name": "gear", + "unicode": "2699-fe0f" + }, + { + "type": "text", + "text": " Spark 3.4 / OL-Spark 1.4.1" + } + ] + } + ] + } + ], + "team": "T01CWUYP5AR", + "thread_ts": "1697527077.180169", + "reply_count": 11, + "reply_users_count": 3, + "latest_reply": "1698655576.941979", + "reply_users": [ + "U02MK6YNAQ5", + "U05T8BJD4DU", + "U05TU0U224A" + ], + "is_locked": false, + "subscribed": false, + "replies": [ + { + "client_msg_id": "fda3c028-90cd-4685-b800-793940aa4314", + "type": "message", + "text": "In general, we assume that OL events per run are cumulative. So, if you have 20 events with the same `runId` , then even if a single event contains some facet, we consider this is OK and let the backend combine it together. That's what we do in Marquez project (a reference backend architecture for OL) and that's why it is worth to use in Marquez as a rest API.\n\nAre you able to use job `namespace` to aggregate all the Spark actions run within the databricks notebook? This is something that should serve this purpose.", + "user": "U02MK6YNAQ5", + "ts": "1697540147.188109", + "blocks": [ + { + "type": "rich_text", + "block_id": "h3dwF", + "elements": [ + { + "type": "rich_text_section", + "elements": [ + { + "type": "text", + "text": "In general, we assume that OL events per run are cumulative. So, if you have 20 events with the same " + }, + { + "type": "text", + "text": "runId", + "style": { + "code": true + } + }, + { + "type": "text", + "text": " , then even if a single event contains some facet, we consider this is OK and let the backend combine it together. That's what we do in Marquez project (a reference backend architecture for OL) and that's why it is worth to use in Marquez as a rest API.\n\nAre you able to use job " + }, + { + "type": "text", + "text": "namespace", + "style": { + "code": true + } + }, + { + "type": "text", + "text": " to aggregate all the Spark actions run within the databricks notebook? This is something that should serve this purpose." + } + ] + } + ] + } + ], + "team": "T01CWUYP5AR", + "thread_ts": "1697527077.180169", + "parent_user_id": "U05TU0U224A" + }, + { + "client_msg_id": "ff65cf2d-b75c-460a-a165-bc36665c4b3f", + "type": "message", + "text": "<@U05TU0U224A> for Spark 3.4 I don't see the environment-properties showing up at all, but if you run the code as it is, register a listener on SparkListenerJobStart and get the properties, all of those properties will show up. There's an event filter that filters out the SparkListenerJobStart, I suspect that filtered out the \"unneccessary\" events.. was trying to do a custom build to do that, but still trying to setup Hadoop and Spark on my local", + "user": "U05T8BJD4DU", + "ts": "1697561313.153019", + "blocks": [ + { + "type": "rich_text", + "block_id": "7A+Ai", + "elements": [ + { + "type": "rich_text_section", + "elements": [ + { + "type": "user", + "user_id": "U05TU0U224A" + }, + { + "type": "text", + "text": " for Spark 3.4 I don't see the environment-properties showing up at all, but if you run the code as it is, register a listener on SparkListenerJobStart and get the properties, all of those properties will show up. There's an event filter that filters out the SparkListenerJobStart, I suspect that filtered out the \"unneccessary\" events.. was trying to do a custom build to do that, but still trying to setup Hadoop and Spark on my local" + } + ] + } + ] + } + ], + "team": "T01CWUYP5AR", + "thread_ts": "1697527077.180169", + "parent_user_id": "U05TU0U224A" + }, + { + "type": "message", + "text": "<@U02MK6YNAQ5> you are right. This is what we are doing as well, combining events with the same runId to process the information on our backend. But even so, there are several runIds without this information. I went through these events to have a better view of what was happening. As you can see from 7 runIds, only 3 were showing the \"environment-properties\" attribute. Some condition is not being met here, or maybe it is what <@U05T8BJD4DU> suspects and there's some sort of filtering of unnecessary events", + "files": [ + { + "id": "F06183ZEM39", + "created": 1697620903, + "timestamp": 1697620903, + "name": "image.png", + "title": "image.png", + "mimetype": "image/png", + "filetype": "png", + "pretty_type": "PNG", + "user": "U05TU0U224A", + "user_team": "T01CWUYP5AR", + "editable": false, + "size": 94544, + "mode": "hosted", + "is_external": false, + "external_type": "", + "is_public": true, + "public_url_shared": false, + "display_as_bot": false, + "username": "", + "url_private": "https://files.slack.com/files-pri/T01CWUYP5AR-F06183ZEM39/image.png", + "url_private_download": "https://files.slack.com/files-pri/T01CWUYP5AR-F06183ZEM39/download/image.png", + "media_display_type": "unknown", + "thumb_64": "https://files.slack.com/files-tmb/T01CWUYP5AR-F06183ZEM39-1bd03f07db/image_64.png", + "thumb_80": "https://files.slack.com/files-tmb/T01CWUYP5AR-F06183ZEM39-1bd03f07db/image_80.png", + "thumb_360": "https://files.slack.com/files-tmb/T01CWUYP5AR-F06183ZEM39-1bd03f07db/image_360.png", + "thumb_360_w": 360, + "thumb_360_h": 94, + "thumb_480": "https://files.slack.com/files-tmb/T01CWUYP5AR-F06183ZEM39-1bd03f07db/image_480.png", + "thumb_480_w": 480, + "thumb_480_h": 125, + "thumb_160": "https://files.slack.com/files-tmb/T01CWUYP5AR-F06183ZEM39-1bd03f07db/image_160.png", + "thumb_720": "https://files.slack.com/files-tmb/T01CWUYP5AR-F06183ZEM39-1bd03f07db/image_720.png", + "thumb_720_w": 720, + "thumb_720_h": 188, + "thumb_800": "https://files.slack.com/files-tmb/T01CWUYP5AR-F06183ZEM39-1bd03f07db/image_800.png", + "thumb_800_w": 800, + "thumb_800_h": 209, + "thumb_960": "https://files.slack.com/files-tmb/T01CWUYP5AR-F06183ZEM39-1bd03f07db/image_960.png", + "thumb_960_w": 960, + "thumb_960_h": 250, + "thumb_1024": "https://files.slack.com/files-tmb/T01CWUYP5AR-F06183ZEM39-1bd03f07db/image_1024.png", + "thumb_1024_w": 1024, + "thumb_1024_h": 267, + "original_w": 1326, + "original_h": 346, + "thumb_tiny": "AwAMADC4CR0Jo3N6mgdKSgCRfmGTz9aaSA5yM05OlIPvmgBynI44pE+5Tqav3aAP/9k=", + "permalink": "https://openlineage.slack.com/files/U05TU0U224A/F06183ZEM39/image.png", + "permalink_public": "https://slack-files.com/T01CWUYP5AR-F06183ZEM39-eb5ae0b512", + "is_starred": false, + "has_rich_preview": false, + "file_access": "visible" + } + ], + "upload": false, + "user": "U05TU0U224A", + "display_as_bot": false, + "ts": "1697620996.359989", + "blocks": [ + { + "type": "rich_text", + "block_id": "VMnXI", + "elements": [ + { + "type": "rich_text_section", + "elements": [ + { + "type": "user", + "user_id": "U02MK6YNAQ5" + }, + { + "type": "text", + "text": " you are right. This is what we are doing as well, combining events with the same runId to process the information on our backend. But even so, there are several runIds without this information. I went through these events to have a better view of what was happening. As you can see from 7 runIds, only 3 were showing the \"environment-properties\" attribute. Some condition is not being met here, or maybe it is what " + }, + { + "type": "user", + "user_id": "U05T8BJD4DU" + }, + { + "type": "text", + "text": " suspects and there's some sort of filtering of unnecessary events" + } + ] + } + ] + } + ], + "edited": { + "user": "U05TU0U224A", + "ts": "1697621162.000000" + }, + "client_msg_id": "5dadc8a4-4dde-4110-bc14-fb6395b9d7fd", + "thread_ts": "1697527077.180169", + "parent_user_id": "U05TU0U224A" + }, + { + "client_msg_id": "60fbeb4e-698f-46a8-8c8f-cbe064f527ed", + "type": "message", + "text": "<@U05TU0U224A>, If you are able to provide a small Spark script such that none of the OL events contain the environment-properties, but at least one should, please raise an issue for this.", + "user": "U02MK6YNAQ5", + "ts": "1697696883.665629", + "blocks": [ + { + "type": "rich_text", + "block_id": "pCmkO", + "elements": [ + { + "type": "rich_text_section", + "elements": [ + { + "type": "user", + "user_id": "U05TU0U224A" + }, + { + "type": "text", + "text": ", If you are able to provide a small Spark script such that none of the OL events contain the environment-properties, but at least one should, please raise an issue for this." + } + ] + } + ] + } + ], + "team": "T01CWUYP5AR", + "thread_ts": "1697527077.180169", + "parent_user_id": "U05TU0U224A" + }, + { + "client_msg_id": "68dff01d-6618-41b9-b079-39d9c6a1b818", + "type": "message", + "text": "It's extremely helpful when community open issues that are not only described well, but also contain small piece of code needed to reproduce this.", + "user": "U02MK6YNAQ5", + "ts": "1697696951.035439", + "blocks": [ + { + "type": "rich_text", + "block_id": "J3sJz", + "elements": [ + { + "type": "rich_text_section", + "elements": [ + { + "type": "text", + "text": "It's extremely helpful when community open issues that are not only described well, but also contain small piece of code needed to reproduce this." + } + ] + } + ] + } + ], + "team": "T01CWUYP5AR", + "thread_ts": "1697527077.180169", + "parent_user_id": "U05TU0U224A" + }, + { + "client_msg_id": "c2342cdb-bf2b-43cb-9773-43e73083028c", + "type": "message", + "text": "I know. that's the goal. that is why I wanted to understand in the first place if there was any condition preventing this from happening, but now i get that this is not expected behaviour.", + "user": "U05TU0U224A", + "ts": "1697698779.304399", + "blocks": [ + { + "type": "rich_text", + "block_id": "versw", + "elements": [ + { + "type": "rich_text_section", + "elements": [ + { + "type": "text", + "text": "I know. that's the goal. that is why I wanted to understand in the first place if there was any condition preventing this from happening, but now i get that this is not expected behaviour." + } + ] + } + ] + } + ], + "team": "T01CWUYP5AR", + "thread_ts": "1697527077.180169", + "parent_user_id": "U05TU0U224A", + "reactions": [ + { + "name": "+1", + "users": [ + "U02MK6YNAQ5" + ], + "count": 1 + } + ] + }, + { + "client_msg_id": "79cc1d29-f02b-41ee-a715-03920f9f4c15", + "type": "message", + "text": "<@U02MK6YNAQ5> <@U05TU0U224A> I am referring to this: ", + "user": "U05T8BJD4DU", + "ts": "1697737440.755279", + "blocks": [ + { + "type": "rich_text", + "block_id": "q5MGb", + "elements": [ + { + "type": "rich_text_section", + "elements": [ + { + "type": "user", + "user_id": "U02MK6YNAQ5" + }, + { + "type": "text", + "text": " " + }, + { + "type": "user", + "user_id": "U05TU0U224A" + }, + { + "type": "text", + "text": " I am referring to this: " + }, + { + "type": "link", + "url": "https://github.com/OpenLineage/OpenLineage/blob/main/integration/spark/shared/src/main/java/io/openlineage/spark/agent/filters/DeltaEventFilter.java#L51" + } + ] + } + ] + } + ], + "team": "T01CWUYP5AR", + "attachments": [ + { + "id": 1, + "footer_icon": "https://slack.github.com/static/img/favicon-neutral.png", + "color": "24292f", + "bot_id": "B01VA0FB340", + "app_unfurl_url": "https://github.com/OpenLineage/OpenLineage/blob/main/integration/spark/shared/src/main/java/io/openlineage/spark/agent/filters/DeltaEventFilter.java#L51", + "is_app_unfurl": true, + "app_id": "A01BP7R4KNY", + "fallback": "", + "text": "```\n || isOnJobStartOrEnd(event);\n```", + "title": "", + "footer": "", + "mrkdwn_in": [ + "text" + ] + } + ], + "thread_ts": "1697527077.180169", + "parent_user_id": "U05TU0U224A" + }, + { + "client_msg_id": "4667b9d1-fe7f-4f5d-93ed-27ce05a8a033", + "type": "message", + "text": "Please note that I am getting the same behavior, no code is needed, Spark 3.4+ won't be generating no matter what. I have been testing the same code for 2 months from this issue: \n\nI tried the code without OL and it worked perfectly, so it is OL filtering out the event for sure. I will try posting the code I use to collect the properties.", + "user": "U05T8BJD4DU", + "ts": "1697741343.557699", + "blocks": [ + { + "type": "rich_text", + "block_id": "KgtDT", + "elements": [ + { + "type": "rich_text_section", + "elements": [ + { + "type": "text", + "text": "Please note that I am getting the same behavior, no code is needed, Spark 3.4+ won't be generating no matter what. I have been testing the same code for 2 months from this issue: " + }, + { + "type": "link", + "url": "https://github.com/OpenLineage/OpenLineage/issues/2124" + }, + { + "type": "text", + "text": "\n\nI tried the code without OL and it worked perfectly, so it is OL filtering out the event for sure. I will try posting the code I use to collect the properties." + } + ] + } + ] + } + ], + "team": "T01CWUYP5AR", + "attachments": [ + { + "id": 1, + "footer_icon": "https://slack.github.com/static/img/favicon-neutral.png", + "ts": 1695498902, + "color": "36a64f", + "bot_id": "B01VA0FB340", + "app_unfurl_url": "https://github.com/OpenLineage/OpenLineage/issues/2124", + "is_app_unfurl": true, + "app_id": "A01BP7R4KNY", + "fallback": "#2124 Same Delta Table not catching the location on write", + "text": "*What is the target system?*\n\nSpark / Databricks\n\n*What kind of integration is this?*\n\n☐ Produces OpenLineage metadata\n☐ Consumes OpenLineage metadata\n☐ Something else\n\n*How should this integration be implemented?*\n\nI am using OL 1.2.2, Azure Databricks Runtime 11.3 LTS. When creating a table writing into a ADLS location, OL won't be able to catch the location of the output. But when I read the same object it will be able to read the location as INPUT.\n\nPlease note I have also tested Databricks Runtime 13.3 LTS, Spark 3.4.1 - it will give correct ADLS location in INPUT but the input will only show up once in a blue moon. Most of the time the inputs and outputs are blank.\n\n```\n \"inputs\": [],\n \"outputs\": []\n```\n\n```\nCREATE OR REPLACE TABLE transactions_adj\nUSING DELTA LOCATION ''\nAS\n SELECT\n household_id,\n basket_id,\n week_no,\n day,\n transaction_time,\n store_id,\n product_id,\n amount_list,\n campaign_coupon_discount,\n manuf_coupon_discount,\n manuf_coupon_match_discount,\n total_coupon_discount,\n instore_discount,\n amount_paid,\n units\n FROM (\n SELECT \n household_id,\n basket_id,\n week_no,\n day,\n transaction_time,\n store_id,\n product_id,\n COALESCE(sales_amount - discount_amount - coupon_discount - coupon_discount_match,0.0) as amount_list,\n CASE \n WHEN COALESCE(coupon_discount_match,0.0) = 0.0 THEN -1 * COALESCE(coupon_discount,0.0) \n ELSE 0.0 \n END as campaign_coupon_discount,\n CASE \n WHEN COALESCE(coupon_discount_match,0.0) != 0.0 THEN -1 * COALESCE(coupon_discount,0.0) \n ELSE 0.0 \n END as manuf_coupon_discount,\n -1 * COALESCE(coupon_discount_match,0.0) as manuf_coupon_match_discount,\n -1 * COALESCE(coupon_discount - coupon_discount_match,0.0) as total_coupon_discount,\n COALESCE(-1 * discount_amount,0.0) as instore_discount,\n COALESCE(sales_amount,0.0) as `amount_paid,`\n quantity as units\n FROM transactions\n );\n```\n\nHere's the COMPLETE event:\n\n```\n\n \"outputs\":[\n {\n \"namespace\":\"dbfs\",\n \"name\":\"/user/hive/warehouse/journey.db/transactions_adj\",\n \"facets\":{\n \"dataSource\":{\n \"_producer\":\"\",\n \"_schemaURL\":\"\",\n \"name\":\"dbfs\",\n \"uri\":\"dbfs\"\n },\n\n```\n\nBelow logical plan shows the path:\n\n```\n== Analyzed Logical Plan ==\nnum_affected_rows: bigint, num_inserted_rows: bigint\nReplaceTableAsSelect TableSpec(Map(),Some(DELTA),Map(),Some(),None,None,false,Set()), true\n:- ResolvedIdentifier com.databricks.sql.managedcatalog.UnityCatalogV2Proxy@6251a8df, default.transactions_adj\n+- Project [household_id#184, basket_id#185L, week_no#193, day#186, transaction_time#192, store_id#190, product_id#187, amount_list#147, campaign_coupon_discount#148, manuf_coupon_discount#149, manuf_coupon_match_discount#150, total_coupon_discount#151, instore_discount#152, amount_paid#153, units#154]\n +- SubqueryAlias __auto_generated_subquery_name\n +- Project [household_id#184, basket_id#185L, week_no#193, day#186, transaction_time#192, store_id#190, product_id#187, coalesce(cast((((sales_amount#189 - discount_amount#191) - coupon_discount#194) - coupon_discount_match#195) as double), cast(0.0 as double)) AS amount_list#147, CASE WHEN (coalesce(cast(coupon_discount_match#195 as double), cast(0.0 as double)) = cast(0.0 as double)) THEN (cast(-1 as double) * coalesce(cast(coupon_discount#194 as double), cast(0.0 as double))) ELSE cast(0.0 as double) END AS campaign_coupon_discount#148, CASE WHEN NOT (coalesce(cast(coupon_discount_match#195 as double), cast(0.0 as double)) = cast(0.0 as double)) THEN (cast(-1 as double) * coalesce(cast(coupon_discount#194 as double), cast(0.0 as double))) ELSE cast(0.0 as double) END AS manuf_coupon_discount#149, (cast(-1 as double) * coalesce(cast(coupon_discount_match#195 as double), cast(0.0 as double))) AS manuf_coupon_match_discount#150, (cast(-1 as double) * coalesce(cast((coupon_discount#194 - coupon_discount_match#195) as double), cast(0.0 as double))) AS total_coupon_discount#151, coalesce(cast((cast(-1 as float) * discount_amount#191) as double), cast(0.0 as double)) AS instore_discount#152, coalesce(cast(sales_amount#189 as double), cast(0.0 as double)) AS amount_paid#153, quantity#188 AS units#154]\n +- SubqueryAlias spark_catalog.default.transactions\n +- Relation spark_catalog.default.transactions[household_id#184,basket_id#185L,day#186,product_id#187,quantity#188,sales_amount#189,store_id#190,discount_amount#191,transaction_time#192,week_no#193,coupon_discount#194,coupon_discount_match#195] parquet\n```\n\n*Where should this integration be implemented?*\n\n☐ In the target system\n☐ In the OpenLineage repo\n☐ Somewhere else\n\n*Do you plan to make this contribution yourself?*\n\n☐ I am interested in doing this work", + "title": "#2124 Same Delta Table not catching the location on write", + "title_link": "https://github.com/OpenLineage/OpenLineage/issues/2124", + "footer": "", + "fields": [ + { + "value": "integration/spark, integration/databricks", + "title": "Labels", + "short": true + }, + { + "value": "3", + "title": "Comments", + "short": true + } + ], + "mrkdwn_in": [ + "text" + ] + } + ], + "thread_ts": "1697527077.180169", + "parent_user_id": "U05TU0U224A" + }, + { + "client_msg_id": "a8e01f77-a721-4e7e-ad4a-377f292c532a", + "type": "message", + "text": "this code proves that the prosperities are still there, somehow got filtered out by OL:\n\n```%scala\nimport org.apache.spark.scheduler._\n\nclass JobStartListener extends SparkListener {\n override def onJobStart(jobStart: SparkListenerJobStart): Unit = {\n // Extract properties here\n val jobId = jobStart.jobId\n val stageInfos = jobStart.stageInfos\n val properties = jobStart.properties\n\n // You can print properties or save them somewhere\n println(s\"JobId: $jobId, Stages: ${stageInfos.size}, Properties: $properties\")\n }\n}\n\nval listener = new JobStartListener()\nspark.sparkContext.addSparkListener(listener)\n\nval df = spark.range(1000).repartition(10)\ndf.count()```", + "user": "U05T8BJD4DU", + "ts": "1697773577.620419", + "blocks": [ + { + "type": "rich_text", + "block_id": "X9SFZ", + "elements": [ + { + "type": "rich_text_section", + "elements": [ + { + "type": "text", + "text": "this code proves that the prosperities are still there, somehow got filtered out by OL:\n\n" + } + ] + }, + { + "type": "rich_text_preformatted", + "elements": [ + { + "type": "text", + "text": "%scala\nimport org.apache.spark.scheduler._\n\nclass JobStartListener extends SparkListener {\n override def onJobStart(jobStart: SparkListenerJobStart): Unit = {\n // Extract properties here\n val jobId = jobStart.jobId\n val stageInfos = jobStart.stageInfos\n val properties = jobStart.properties\n\n // You can print properties or save them somewhere\n println(s\"JobId: $jobId, Stages: ${stageInfos.size}, Properties: $properties\")\n }\n}\n\nval listener = new JobStartListener()\nspark.sparkContext.addSparkListener(listener)\n\nval df = spark.range(1000).repartition(10)\ndf.count()" + } + ], + "border": 0 + } + ] + } + ], + "team": "T01CWUYP5AR", + "thread_ts": "1697527077.180169", + "parent_user_id": "U05TU0U224A" + }, + { + "client_msg_id": "ea8f6b5d-4d94-4275-b07e-cb72675cda7e", + "type": "message", + "text": "of course feel free to test this logic as well, it still works -- if not the filtering:\n\n", + "user": "U05T8BJD4DU", + "ts": "1697774105.261429", + "blocks": [ + { + "type": "rich_text", + "block_id": "d99Zk", + "elements": [ + { + "type": "rich_text_section", + "elements": [ + { + "type": "text", + "text": "of course feel free to test this logic as well, it still works -- if not the filtering:\n\n" + }, + { + "type": "link", + "url": "https://github.com/OpenLineage/OpenLineage/blob/main/integration/spark/shared/src/main/java/io/openlineage/spark/agent/facets/builder/DatabricksEnvironmentFacetBuilder.java", + "text": "https://github.com/OpenLineage/OpenLineage/blob/main/integration/spark/shared/src/[…]ark/agent/facets/builder/DatabricksEnvironmentFacetBuilder.java" + } + ] + } + ] + } + ], + "team": "T01CWUYP5AR", + "edited": { + "user": "U05T8BJD4DU", + "ts": "1697774126.000000" + }, + "attachments": [ + { + "id": 1, + "footer_icon": "https://slack.github.com/static/img/favicon-neutral.png", + "color": "24292f", + "bot_id": "B01VA0FB340", + "app_unfurl_url": "https://github.com/OpenLineage/OpenLineage/blob/main/integration/spark/shared/src/main/java/io/openlineage/spark/agent/facets/builder/DatabricksEnvironmentFacetBuilder.java", + "is_app_unfurl": true, + "app_id": "A01BP7R4KNY", + "fallback": "", + "text": "```\n/*\n/* Copyright 2018-2023 contributors to the OpenLineage project\n/* SPDX-License-Identifier: Apache-2.0\n*/\n\npackage io.openlineage.spark.agent.facets.builder;\n\nimport com.databricks.backend.daemon.dbutils.MountInfo;\nimport com.databricks.dbutils_v1.DbfsUtils;\nimport io.openlineage.spark.agent.facets.EnvironmentFacet;\nimport io.openlineage.spark.agent.models.DatabricksMountpoint;\nimport io.openlineage.spark.api.CustomFacetBuilder;\nimport io.openlineage.spark.api.OpenLineageContext;\nimport java.lang.reflect.Constructor;\nimport java.lang.reflect.InvocationTargetException;\nimport java.lang.reflect.Parameter;\nimport java.util.ArrayList;\nimport java.util.Arrays;\nimport java.util.HashMap;\nimport java.util.List;\nimport java.util.Map;\nimport java.util.Optional;\nimport java.util.function.BiConsumer;\nimport lombok.extern.slf4j.Slf4j;\nimport org.apache.spark.scheduler.SparkListenerJobStart;\nimport scala.collection.JavaConversions;\n\n/**\n * {@link CustomFacetBuilder} that generates a {@link EnvironmentFacet} when using OpenLineage on\n * Databricks.\n */\n@Slf4j\npublic class DatabricksEnvironmentFacetBuilder\n extends CustomFacetBuilder {\n private Map dbProperties;\n private Class dbutilsClass;\n private DbfsUtils dbutils;\n\n public static boolean isDatabricksRuntime() {\n return System.getenv().containsKey(\"DATABRICKS_RUNTIME_VERSION\");\n }\n\n public DatabricksEnvironmentFacetBuilder() {}\n\n public DatabricksEnvironmentFacetBuilder(OpenLineageContext openLineageContext) {\n dbProperties = new HashMap<>();\n // extract some custom environment variables if needed\n openLineageContext\n .getCustomEnvironmentVariables()\n .ifPresent(\n envVars ->\n envVars.forEach(envVar -> dbProperties.put(envVar, System.getenv().get(envVar))));\n }\n\n @Override\n protected void build(\n SparkListenerJobStart event, BiConsumer consumer) {\n consumer.accept(\n \"environment-properties\",\n new EnvironmentFacet(getDatabricksEnvironmentalAttributes(event)));\n }\n\n private Map getDatabricksEnvironmentalAttributes(SparkListenerJobStart jobStart) {\n if (dbProperties == null) {\n dbProperties = new HashMap<>();\n }\n\n // These are useful properties to extract if they are available\n List dbPropertiesKeys =\n Arrays.asList(\n \"orgId\",\n \"spark.databricks.clusterUsageTags.clusterOwnerOrgId\",\n \"spark.databricks.notebook.path\",\n \"spark.databricks.job.type\",\n \"spark.databricks.job.id\",\n \"spark.databricks.job.runId\",\n \"user\",\n \"userId\",\n \"spark.databricks.clusterUsageTags.clusterName\",\n \"spark.databricks.clusterUsageTags.clusterAllTags\",\n \"spark.databricks.clusterUsageTags.azureSubscriptionId\");\n dbPropertiesKeys.stream()\n .forEach(\n (p) -> {\n dbProperties.put(p, jobStart.properties().getProperty(p));\n });\n\n /**\n * Azure Databricks makes available a dbutils mount point to list aliased paths to cloud\n * storage. However, that dbutils object is not available inside a spark listener. We must\n * access it via reflection.\n */\n try {\n Optional dbfsUtils = getDbfsUtils();\n if (!dbfsUtils.isPresent()) {\n dbProperties.put(\"mountPoints\", new ArrayList());\n } else {\n dbProperties.put(\"mountPoints\", getDatabricksMountpoints(dbfsUtils.get()));\n }\n\n } catch (Exception e) {\n log.warn(\"Failed to load dbutils in OpenLineageListener:\", e);\n dbProperties.put(\"mountPoints\", new ArrayList());\n }\n return dbProperties;\n }\n\n // Starting in Databricks Runtime 11, there is a new constructor for DbFsUtils\n // If running on an older version, the constructor has no parameters.\n // If running on DBR 11 or above, you need to specify whether you allow mount operations (true or\n // false)\n private static Optional getDbfsUtils()\n throws ClassNotFoundException, InstantiationException, IllegalAccessException,\n IllegalArgumentException, InvocationTargetException {\n Class dbutilsClass = Class.forName(\"com.databricks.dbutils_v1.impl.DbfsUtilsImpl\");\n Constructor[] dbutilsConstructors = dbutilsClass.getDeclaredConstructors();\n if (dbutilsConstructors.length == 0) {\n log.warn(\n \"Failed to load dbutils in OpenLineageListener as there were no declared constructors\");\n return Optional.empty();\n }\n Constructor firstConstructor = dbutilsConstructors[0];\n Parameter[] constructorParams = firstConstructor.getParameters();\n if (constructorParams.length == 0) {\n log.debug(\"DbUtils constructor had no parameters\");\n return Optional.of((DbfsUtils) firstConstructor.newInstance());\n } else if (constructorParams.length == 1\n && constructorParams[0].getName().equals(\"allowMountOperations\")) {\n log.debug(\"DbUtils constructor had one parameter named allowMountOperations\");\n return Optional.of((DbfsUtils) firstConstructor.newInstance(true));\n } else {\n log.warn(\n \"dbutils had {} constructors and the first constructor had {} params\",\n dbutilsConstructors.length,\n constructorParams.length);\n return Optional.empty();\n }\n }\n\n private static List getDatabricksMountpoints(DbfsUtils dbutils) {\n List mountpoints = new ArrayList<>();\n List mountsList = JavaConversions.seqAsJavaList(dbutils.mounts());\n for (MountInfo mount : mountsList) {\n mountpoints.add(new DatabricksMountpoint(mount.mountPoint(), mount.source()));\n }\n return mountpoints;\n }\n}\n\n```", + "title": "", + "footer": "", + "mrkdwn_in": [ + "text" + ] + } + ], + "thread_ts": "1697527077.180169", + "parent_user_id": "U05TU0U224A" + }, + { + "client_msg_id": "5541ebd3-b058-4421-854a-69da0f2fadb5", + "type": "message", + "text": "Any ideas on how could i test it?", + "user": "U05TU0U224A", + "ts": "1698655576.941979", + "blocks": [ + { + "type": "rich_text", + "block_id": "/YRlt", + "elements": [ + { + "type": "rich_text_section", + "elements": [ + { + "type": "text", + "text": "Any ideas on how could i test it?" + } + ] + } + ] + } + ], + "team": "T01CWUYP5AR", + "thread_ts": "1697527077.180169", + "parent_user_id": "U05TU0U224A" + } + ] + }, + { + "client_msg_id": "fe0af91b-6ec2-4d34-b71e-aea9c88ce2c4", + "type": "message", + "text": "Hi team, I am running the following pyspark code in a cell:\n```print(\"SELECTING 100 RECORDS FROM METADATA TABLE\")\ndf = spark.sql(\"\"\"select * from limit 100\"\"\")\n\nprint(\"WRITING (1) 100 RECORDS FROM METADATA TABLE\")\ndf.write.mode(\"overwrite\").format('delta').save(\"\")\ndf.createOrReplaceTempView(\"temp_metadata\")\n\nprint(\"WRITING (2) 100 RECORDS FROM METADATA TABLE\")\ndf.write.mode(\"overwrite\").format(\"delta\").save(\"\")\n\nprint(\"READING (1) 100 RECORDS FROM METADATA TABLE\")\ndf_read = spark.read.format('delta').load(\"\")\ndf_read.createOrReplaceTempView(\"metadata_1\")\n\nprint(\"DOING THE MERGE INTO SQL STEP!\")\ndf_new = spark.sql(\"\"\"\n MERGE INTO metadata_1\n USING
\n ON metadata_1.id = temp_metadata.id\n WHEN MATCHED THEN UPDATE SET \n metadata_1.id = temp_metadata.id,\n metadata_1.aspect = temp_metadata.aspect\n WHEN NOT MATCHED THEN INSERT (id, aspect) \n VALUES (temp_metadata.id, temp_metadata.aspect)\n\"\"\")```\nI am running with debug log levels. I actually don't see any of the events being logged for `SaveIntoDataSourceCommand` or the `MergeIntoCommand`, but OL is in fact emitting events to the backend. It seems like the events are just not being logged... I actually observe this for all delta table related spark sql queries...", + "user": "U04EZ2LPDV4", + "ts": "1697179720.032079", + "blocks": [ + { + "type": "rich_text", + "block_id": "NqsvT", + "elements": [ + { + "type": "rich_text_section", + "elements": [ + { + "type": "text", + "text": "Hi team, I am running the following pyspark code in a cell:\n" + } + ] + }, + { + "type": "rich_text_preformatted", + "elements": [ + { + "type": "text", + "text": "print(\"SELECTING 100 RECORDS FROM METADATA TABLE\")\ndf = spark.sql(\"\"\"select * from
limit 100\"\"\")\n\nprint(\"WRITING (1) 100 RECORDS FROM METADATA TABLE\")\ndf.write.mode(\"overwrite\").format('delta').save(\"\")\ndf.createOrReplaceTempView(\"temp_metadata\")\n\nprint(\"WRITING (2) 100 RECORDS FROM METADATA TABLE\")\ndf.write.mode(\"overwrite\").format(\"delta\").save(\"\")\n\nprint(\"READING (1) 100 RECORDS FROM METADATA TABLE\")\ndf_read = spark.read.format('delta').load(\"\")\ndf_read.createOrReplaceTempView(\"metadata_1\")\n\nprint(\"DOING THE MERGE INTO SQL STEP!\")\ndf_new = spark.sql(\"\"\"\n MERGE INTO metadata_1\n USING
\n ON metadata_1.id = temp_metadata.id\n WHEN MATCHED THEN UPDATE SET \n metadata_1.id = temp_metadata.id,\n metadata_1.aspect = temp_metadata.aspect\n WHEN NOT MATCHED THEN INSERT (id, aspect) \n VALUES (temp_metadata.id, temp_metadata.aspect)\n\"\"\")" + } + ], + "border": 0 + }, + { + "type": "rich_text_section", + "elements": [ + { + "type": "text", + "text": "I am running with debug log levels. I actually don't see any of the events being logged for " + }, + { + "type": "text", + "text": "SaveIntoDataSourceCommand", + "style": { + "code": true + } + }, + { + "type": "text", + "text": " or the " + }, + { + "type": "text", + "text": "MergeIntoCommand", + "style": { + "code": true + } + }, + { + "type": "text", + "text": ", but OL is in fact emitting events to the backend. It seems like the events are just not being logged... I actually observe this for all delta table related spark sql queries..." + } + ] + } + ] + } + ], + "team": "T01CWUYP5AR", + "edited": { + "user": "U04EZ2LPDV4", + "ts": "1697428845.000000" + }, + "thread_ts": "1697179720.032079", + "reply_count": 6, + "reply_users_count": 2, + "latest_reply": "1698310610.069939", + "reply_users": [ + "U04EZ2LPDV4", + "U05FLJE4GDU" + ], + "is_locked": false, + "subscribed": false, + "replies": [ + { + "client_msg_id": "2192ef5f-6edb-41ea-ab67-74a24936b1e5", + "type": "message", + "text": "Hi <@U02MK6YNAQ5> is this expected? CMIIW but we should expect to see the events being logged when running with debug log level right?", + "user": "U04EZ2LPDV4", + "ts": "1697428902.144169", + "blocks": [ + { + "type": "rich_text", + "block_id": "MjNBU", + "elements": [ + { + "type": "rich_text_section", + "elements": [ + { + "type": "text", + "text": "Hi " + }, + { + "type": "user", + "user_id": "U02MK6YNAQ5" + }, + { + "type": "text", + "text": " is this expected? CMIIW but we should expect to see the events being logged when running with debug log level right?" + } + ] + } + ] + } + ], + "team": "T01CWUYP5AR", + "thread_ts": "1697179720.032079", + "parent_user_id": "U04EZ2LPDV4" + }, + { + "client_msg_id": "33219d08-21e7-4f93-91a5-7003dc0afe9c", + "type": "message", + "text": "It's impossible to know without seeing how you've configured the listener.\n\nCan you show this configuration?", + "user": "U05FLJE4GDU", + "ts": "1697444250.594509", + "blocks": [ + { + "type": "rich_text", + "block_id": "cDgl4", + "elements": [ + { + "type": "rich_text_section", + "elements": [ + { + "type": "text", + "text": "It's impossible to know without seeing how you've configured the listener.\n\nCan you show this configuration?" + } + ] + } + ] + } + ], + "team": "T01CWUYP5AR", + "thread_ts": "1697179720.032079", + "parent_user_id": "U04EZ2LPDV4" + }, + { + "client_msg_id": "8645ab34-2922-4e1b-9a0f-993e64435b62", + "type": "message", + "text": "```spark.openlineage.transport.url <url>\nspark.openlineage.transport.endpoint /<endpoint>\nspark.openlineage.transport.type http\nspark.extraListeners io.openlineage.spark.agent.OpenLineageSparkListener\nspark.openlineage.facets.custom_environment_variables [BUNCH_OF_VARIABLES;]\nspark.openlineage.facets.disabled [spark_unknown\\;spark.logicalPlan]```\nThese are my spark configs... I'm setting log level to debug with `sc.setLogLevel(\"DEBUG\")`", + "user": "U04EZ2LPDV4", + "ts": "1697526920.491239", + "blocks": [ + { + "type": "rich_text", + "block_id": "NF5xw", + "elements": [ + { + "type": "rich_text_preformatted", + "elements": [ + { + "type": "text", + "text": "spark.openlineage.transport.url \nspark.openlineage.transport.endpoint /\nspark.openlineage.transport.type http\nspark.extraListeners io.openlineage.spark.agent.OpenLineageSparkListener\nspark.openlineage.facets.custom_environment_variables [BUNCH_OF_VARIABLES;]\nspark.openlineage.facets.disabled [spark_unknown\\;spark.logicalPlan]" + } + ], + "border": 0 + }, + { + "type": "rich_text_section", + "elements": [ + { + "type": "text", + "text": "These are my spark configs... I'm setting log level to debug with " + }, + { + "type": "text", + "text": "sc.setLogLevel(\"DEBUG\")", + "style": { + "code": true + } + } + ] + } + ] + } + ], + "team": "T01CWUYP5AR", + "thread_ts": "1697179720.032079", + "parent_user_id": "U04EZ2LPDV4" + }, + { + "client_msg_id": "b7b05cd0-b076-4719-87cc-84ae866cedaf", + "type": "message", + "text": "Two things:\n\n1. If you want debug logs, you're going to have to provide a `log4j.properties` file or `log4j2.properties` file depending on the version of spark you're running. In that file, you will need to configure the logging levels. If I am not mistaken, the `sc.setLogLevel` controls ONLY the log levels of Spark namespaced components (i.e., `org.apache.spark`)\n2. You're telling the listener to emit to a URL. If you want to see the events emitted to the console, then set `spark.openlineage.transport.type=console`, and remove the other `spark.openlineage.transport.*` configurations.\nDo either (1) or (2).", + "user": "U05FLJE4GDU", + "ts": "1697532003.340519", + "blocks": [ + { + "type": "rich_text", + "block_id": "BJmP5", + "elements": [ + { + "type": "rich_text_section", + "elements": [ + { + "type": "text", + "text": "Two things:\n\n" + } + ] + }, + { + "type": "rich_text_list", + "elements": [ + { + "type": "rich_text_section", + "elements": [ + { + "type": "text", + "text": "If you want debug logs, you're going to have to provide a " + }, + { + "type": "text", + "text": "log4j.properties", + "style": { + "code": true + } + }, + { + "type": "text", + "text": " file or " + }, + { + "type": "text", + "text": "log4j2.properties", + "style": { + "code": true + } + }, + { + "type": "text", + "text": " file depending on the version of spark you're running. In that file, you will need to configure the logging levels. If I am not mistaken, the " + }, + { + "type": "text", + "text": "sc.setLogLevel", + "style": { + "code": true + } + }, + { + "type": "text", + "text": " controls ONLY the log levels of Spark namespaced components (i.e., " + }, + { + "type": "text", + "text": "org.apache.spark", + "style": { + "code": true + } + }, + { + "type": "text", + "text": ")" + } + ] + }, + { + "type": "rich_text_section", + "elements": [ + { + "type": "text", + "text": "You're telling the listener to emit to a URL. If you want to see the events emitted to the console, then set " + }, + { + "type": "text", + "text": "spark.openlineage.transport.type=console", + "style": { + "code": true + } + }, + { + "type": "text", + "text": ", and remove the other " + }, + { + "type": "text", + "text": "spark.openlineage.transport.*", + "style": { + "code": true + } + }, + { + "type": "text", + "text": " configurations." + } + ] + } + ], + "style": "ordered", + "indent": 0, + "border": 0 + }, + { + "type": "rich_text_section", + "elements": [ + { + "type": "text", + "text": "\nDo either (1) or (2)." + } + ] + } + ] + } + ], + "team": "T01CWUYP5AR", + "thread_ts": "1697179720.032079", + "parent_user_id": "U04EZ2LPDV4" + }, + { + "client_msg_id": "b78fafde-258c-4317-bb95-8dca99235ab6", + "type": "message", + "text": "<@U05FLJE4GDU> Hi, sflr.\n1. So enabling `sc.setLogLevel` does actually enable debug logs from Openlineage. I can see the events and everyting being logged if I save it as a parquet format instead of delta. \n2. I do want to emit events to the url. But, I would like to just see what exactly are the events being emitted for some specific jobs, since I see that the lineage is incorrect for some MergeInto cases", + "user": "U04EZ2LPDV4", + "ts": "1697777385.439859", + "blocks": [ + { + "type": "rich_text", + "block_id": "g8Thq", + "elements": [ + { + "type": "rich_text_section", + "elements": [ + { + "type": "user", + "user_id": "U05FLJE4GDU" + }, + { + "type": "text", + "text": " Hi, sflr.\n" + } + ] + }, + { + "type": "rich_text_list", + "elements": [ + { + "type": "rich_text_section", + "elements": [ + { + "type": "text", + "text": "So enabling " + }, + { + "type": "text", + "text": "sc.setLogLevel", + "style": { + "code": true + } + }, + { + "type": "text", + "text": " does actually enable debug logs from Openlineage. I can see the events and everyting being logged if I save it as a parquet format instead of delta. " + } + ] + }, + { + "type": "rich_text_section", + "elements": [ + { + "type": "text", + "text": "I do want to emit events to the url. But, I would like to just see what exactly are the events being emitted for some specific jobs, since I see that the lineage is incorrect for some MergeInto cases" + } + ] + } + ], + "style": "ordered", + "indent": 0, + "border": 0 + } + ] + } + ], + "team": "T01CWUYP5AR", + "edited": { + "user": "U04EZ2LPDV4", + "ts": "1697777397.000000" + }, + "thread_ts": "1697179720.032079", + "parent_user_id": "U04EZ2LPDV4" + }, + { + "client_msg_id": "11b61616-c248-4409-a823-38e839f30dca", + "type": "message", + "text": "Hi <@U05FLJE4GDU> would like to check again on whether you'd have any thoughts about this... Thanks! :slightly_smiling_face:", + "user": "U04EZ2LPDV4", + "ts": "1698310610.069939", + "blocks": [ + { + "type": "rich_text", + "block_id": "eKcmt", + "elements": [ + { + "type": "rich_text_section", + "elements": [ + { + "type": "text", + "text": "Hi " + }, + { + "type": "user", + "user_id": "U05FLJE4GDU" + }, + { + "type": "text", + "text": " would like to check again on whether you'd have any thoughts about this... Thanks! " + }, + { + "type": "emoji", + "name": "slightly_smiling_face", + "unicode": "1f642" + } + ] + } + ] + } + ], + "team": "T01CWUYP5AR", + "thread_ts": "1697179720.032079", + "parent_user_id": "U04EZ2LPDV4" + } + ] + }, + { + "client_msg_id": "7e2e374d-b98a-475c-80af-541046c52476", + "type": "message", + "text": "This might be a dumb question, I guess I need to setup local Spark in order for the Spark tests to run successfully?", + "user": "U05T8BJD4DU", + "ts": "1697137714.503349", + "blocks": [ + { + "type": "rich_text", + "block_id": "U1OfA", + "elements": [ + { + "type": "rich_text_section", + "elements": [ + { + "type": "text", + "text": "This might be a dumb question, I guess I need to setup local Spark in order for the Spark tests to run successfully?" + } + ] + } + ] + } + ], + "team": "T01CWUYP5AR", + "thread_ts": "1697137714.503349", + "reply_count": 15, + "reply_users_count": 3, + "latest_reply": "1698025583.327319", + "reply_users": [ + "U02MK6YNAQ5", + "U05QL7LN2GH", + "U05T8BJD4DU" + ], + "is_locked": false, + "subscribed": false, + "replies": [ + { + "client_msg_id": "d14cb78f-6549-4460-a171-fb04e26bb044", + "type": "message", + "text": "just follow these instructions: ", + "user": "U02MK6YNAQ5", + "ts": "1697176579.973179", + "blocks": [ + { + "type": "rich_text", + "block_id": "ouUqx", + "elements": [ + { + "type": "rich_text_section", + "elements": [ + { + "type": "text", + "text": "just follow these instructions: " + }, + { + "type": "link", + "url": "https://github.com/OpenLineage/OpenLineage/tree/main/integration/spark#build" + } + ] + } + ] + } + ], + "team": "T01CWUYP5AR", + "thread_ts": "1697137714.503349", + "parent_user_id": "U05T8BJD4DU" + }, + { + "client_msg_id": "e6dcc9be-50b2-408c-ae02-6f4072d94a2e", + "type": "message", + "text": "when trying to install openlineage-java in local via this command --> cd ../../client/java/ && ./gradlew publishToMavenLocal, i am receiving this error\n```> Task :signMavenJavaPublication FAILED\n\nFAILURE: Build failed with an exception.\n\n* What went wrong:\nExecution failed for task ':signMavenJavaPublication'.\n> Cannot perform signing task ':signMavenJavaPublication' because it has no configured signatory```", + "user": "U05QL7LN2GH", + "ts": "1697193716.992739", + "blocks": [ + { + "type": "rich_text", + "block_id": "Ebo87", + "elements": [ + { + "type": "rich_text_section", + "elements": [ + { + "type": "text", + "text": "when trying to install openlineage-java in local via this command --> cd ../../client/java/ && ./gradlew publishToMavenLocal, i am receiving this error\n" + } + ] + }, + { + "type": "rich_text_preformatted", + "elements": [ + { + "type": "text", + "text": "> Task :signMavenJavaPublication FAILED\n\nFAILURE: Build failed with an exception.\n\n* What went wrong:\nExecution failed for task ':signMavenJavaPublication'.\n> Cannot perform signing task ':signMavenJavaPublication' because it has no configured signatory" + } + ], + "border": 0 + } + ] + } + ], + "team": "T01CWUYP5AR", + "thread_ts": "1697137714.503349", + "parent_user_id": "U05T8BJD4DU" + }, + { + "type": "message", + "text": "<@U02MK6YNAQ5> this is what I am getting", + "files": [ + { + "id": "F061S5ZMF08", + "created": 1697218504, + "timestamp": 1697218504, + "name": "image.png", + "title": "image.png", + "mimetype": "image/png", + "filetype": "png", + "pretty_type": "PNG", + "user": "U05T8BJD4DU", + "user_team": "T01CWUYP5AR", + "editable": false, + "size": 107167, + "mode": "hosted", + "is_external": false, + "external_type": "", + "is_public": true, + "public_url_shared": false, + "display_as_bot": false, + "username": "", + "url_private": "https://files.slack.com/files-pri/T01CWUYP5AR-F061S5ZMF08/image.png", + "url_private_download": "https://files.slack.com/files-pri/T01CWUYP5AR-F061S5ZMF08/download/image.png", + "media_display_type": "unknown", + "thumb_64": "https://files.slack.com/files-tmb/T01CWUYP5AR-F061S5ZMF08-85392c9570/image_64.png", + "thumb_80": "https://files.slack.com/files-tmb/T01CWUYP5AR-F061S5ZMF08-85392c9570/image_80.png", + "thumb_360": "https://files.slack.com/files-tmb/T01CWUYP5AR-F061S5ZMF08-85392c9570/image_360.png", + "thumb_360_w": 360, + "thumb_360_h": 60, + "thumb_480": "https://files.slack.com/files-tmb/T01CWUYP5AR-F061S5ZMF08-85392c9570/image_480.png", + "thumb_480_w": 480, + "thumb_480_h": 80, + "thumb_160": "https://files.slack.com/files-tmb/T01CWUYP5AR-F061S5ZMF08-85392c9570/image_160.png", + "thumb_720": "https://files.slack.com/files-tmb/T01CWUYP5AR-F061S5ZMF08-85392c9570/image_720.png", + "thumb_720_w": 720, + "thumb_720_h": 120, + "thumb_800": "https://files.slack.com/files-tmb/T01CWUYP5AR-F061S5ZMF08-85392c9570/image_800.png", + "thumb_800_w": 800, + "thumb_800_h": 133, + "thumb_960": "https://files.slack.com/files-tmb/T01CWUYP5AR-F061S5ZMF08-85392c9570/image_960.png", + "thumb_960_w": 960, + "thumb_960_h": 160, + "thumb_1024": "https://files.slack.com/files-tmb/T01CWUYP5AR-F061S5ZMF08-85392c9570/image_1024.png", + "thumb_1024_w": 1024, + "thumb_1024_h": 171, + "original_w": 2779, + "original_h": 463, + "thumb_tiny": "AwAHADCpkelJx6UUlABSZoooAM0A80lKOtAH/9k=", + "permalink": "https://openlineage.slack.com/files/U05T8BJD4DU/F061S5ZMF08/image.png", + "permalink_public": "https://slack-files.com/T01CWUYP5AR-F061S5ZMF08-b83e34fc5b", + "is_starred": false, + "has_rich_preview": false, + "file_access": "visible" + } + ], + "upload": false, + "user": "U05T8BJD4DU", + "display_as_bot": false, + "ts": "1697218506.981419", + "blocks": [ + { + "type": "rich_text", + "block_id": "d1Buh", + "elements": [ + { + "type": "rich_text_section", + "elements": [ + { + "type": "user", + "user_id": "U02MK6YNAQ5" + }, + { + "type": "text", + "text": " this is what I am getting" + } + ] + } + ] + } + ], + "client_msg_id": "a357d38b-8e93-4925-a1b4-0713d750b564", + "thread_ts": "1697137714.503349", + "parent_user_id": "U05T8BJD4DU" + }, + { + "type": "message", + "text": "attaching the html", + "files": [ + { + "id": "F061FR39UE5", + "created": 1697218554, + "timestamp": 1697218554, + "name": "io.openlineage.spark.agent.lifecycle.plan.AlterTableAddPartitionCommandVisitorTest.html", + "title": "io.openlineage.spark.agent.lifecycle.plan.AlterTableAddPartitionCommandVisitorTest.html", + "mimetype": "text/plain", + "filetype": "html", + "pretty_type": "HTML", + "user": "U05T8BJD4DU", + "user_team": "T01CWUYP5AR", + "editable": true, + "size": 17191, + "mode": "snippet", + "is_external": false, + "external_type": "", + "is_public": true, + "public_url_shared": false, + "display_as_bot": false, + "username": "", + "url_private": "https://files.slack.com/files-pri/T01CWUYP5AR-F061FR39UE5/io.openlineage.spark.agent.lifecycle.plan.altertableaddpartitioncommandvisitortest.html", + "url_private_download": "https://files.slack.com/files-pri/T01CWUYP5AR-F061FR39UE5/download/io.openlineage.spark.agent.lifecycle.plan.altertableaddpartitioncommandvisitortest.html", + "permalink": "https://openlineage.slack.com/files/U05T8BJD4DU/F061FR39UE5/io.openlineage.spark.agent.lifecycle.plan.altertableaddpartitioncommandvisitortest.html", + "permalink_public": "https://slack-files.com/T01CWUYP5AR-F061FR39UE5-b2ba858587", + "edit_link": "https://openlineage.slack.com/files/U05T8BJD4DU/F061FR39UE5/io.openlineage.spark.agent.lifecycle.plan.altertableaddpartitioncommandvisitortest.html/edit", + "preview": "\r\n\r\n\r\n\r\n\r", + "preview_highlight": "
\n
\n
<!DOCTYPE html>
\n
<html>
\n
<head>
\n
<meta http-equiv="Content-Type" content="text/html; charset=utf-8"/>
\n
<meta http-equiv="x-ua-compatible" content="IE=edge"/>
\n
\n
\n", + "lines": 244, + "lines_more": 239, + "preview_is_truncated": true, + "is_starred": false, + "has_rich_preview": false, + "file_access": "visible" + } + ], + "upload": false, + "user": "U05T8BJD4DU", + "display_as_bot": false, + "ts": "1697218560.627389", + "blocks": [ + { + "type": "rich_text", + "block_id": "mRJBE", + "elements": [ + { + "type": "rich_text_section", + "elements": [ + { + "type": "text", + "text": "attaching the html" + } + ] + } + ] + } + ], + "client_msg_id": "69b264e3-7415-433b-bcad-a449fe6d4fc5", + "thread_ts": "1697137714.503349", + "parent_user_id": "U05T8BJD4DU" + }, + { + "client_msg_id": "0b50bf9f-40c4-4bf5-96e8-41be9cf2c202", + "type": "message", + "text": "which java are you using? what is your operation system (is it windows?)?", + "user": "U02MK6YNAQ5", + "ts": "1697439733.216699", + "blocks": [ + { + "type": "rich_text", + "block_id": "/d7dd", + "elements": [ + { + "type": "rich_text_section", + "elements": [ + { + "type": "text", + "text": "which java are you using? what is your operation system (is it windows?)?" + } + ] + } + ] + } + ], + "team": "T01CWUYP5AR", + "thread_ts": "1697137714.503349", + "parent_user_id": "U05T8BJD4DU" + }, + { + "client_msg_id": "ade3fbfa-0b80-4eb6-99d3-07b6e59ffdaf", + "type": "message", + "text": "yes it is Windows, i downloaded java 8 but I can try to build it with Linux subsystem or Mac", + "user": "U05T8BJD4DU", + "ts": "1697441718.743329", + "blocks": [ + { + "type": "rich_text", + "block_id": "rdTqb", + "elements": [ + { + "type": "rich_text_section", + "elements": [ + { + "type": "text", + "text": "yes it is Windows, i downloaded java 8 but I can try to build it with Linux subsystem or Mac" + } + ] + } + ] + } + ], + "team": "T01CWUYP5AR", + "thread_ts": "1697137714.503349", + "parent_user_id": "U05T8BJD4DU" + }, + { + "client_msg_id": "ccf03d2b-686c-46d0-ae2e-dc2ddbe80330", + "type": "message", + "text": "In my case it is Mac", + "user": "U05QL7LN2GH", + "ts": "1697441751.675019", + "blocks": [ + { + "type": "rich_text", + "block_id": "TQvZ2", + "elements": [ + { + "type": "rich_text_section", + "elements": [ + { + "type": "text", + "text": "In my case it is Mac" + } + ] + } + ] + } + ], + "team": "T01CWUYP5AR", + "thread_ts": "1697137714.503349", + "parent_user_id": "U05T8BJD4DU" + }, + { + "client_msg_id": "167256fa-fa11-4a98-a407-da6ece5878be", + "type": "message", + "text": "* Where:\nBuild file '/mnt/c/Users/jason/Downloads/github/OpenLineage/integration/spark/build.gradle' line: 9\n\n* What went wrong:\nAn exception occurred applying plugin request [id: 'com.adarshr.test-logger', version: '3.2.0']\n> Failed to apply plugin [id 'com.adarshr.test-logger']\n > Could not generate a proxy class for class com.adarshr.gradle.testlogger.TestLoggerExtension.\n\n* Try:", + "user": "U05T8BJD4DU", + "ts": "1697442969.487219", + "blocks": [ + { + "type": "rich_text", + "block_id": "oi3kW", + "elements": [ + { + "type": "rich_text_section", + "elements": [ + { + "type": "text", + "text": "* Where:\nBuild file '/mnt/c/Users/jason/Downloads/github/OpenLineage/integration/spark/build.gradle' line: 9\n\n* What went wrong:\nAn exception occurred applying plugin request [id: 'com.adarshr.test-logger', version: '3.2.0']\n> Failed to apply plugin [id 'com.adarshr.test-logger']\n > Could not generate a proxy class for class com.adarshr.gradle.testlogger.TestLoggerExtension.\n\n* Try:" + } + ] + } + ] + } + ], + "team": "T01CWUYP5AR", + "thread_ts": "1697137714.503349", + "parent_user_id": "U05T8BJD4DU" + }, + { + "client_msg_id": "9a25cde0-5748-40eb-a40c-ca6f7cd93514", + "type": "message", + "text": "tried with Linux subsystem", + "user": "U05T8BJD4DU", + "ts": "1697442983.551079", + "blocks": [ + { + "type": "rich_text", + "block_id": "Knyq2", + "elements": [ + { + "type": "rich_text_section", + "elements": [ + { + "type": "text", + "text": "tried with Linux subsystem" + } + ] + } + ] + } + ], + "team": "T01CWUYP5AR", + "thread_ts": "1697137714.503349", + "parent_user_id": "U05T8BJD4DU" + }, + { + "client_msg_id": "aabbc5df-cec7-4979-aca6-e0a716b7d7e9", + "type": "message", + "text": "we don't have any restrictions for windows builds, however it is something we don't test regularly. 2h ago we did have a successful build on circle CI ", + "user": "U02MK6YNAQ5", + "ts": "1697443469.020679", + "blocks": [ + { + "type": "rich_text", + "block_id": "GfkaO", + "elements": [ + { + "type": "rich_text_section", + "elements": [ + { + "type": "text", + "text": "we don't have any restrictions for windows builds, however it is something we don't test regularly. 2h ago we did have a successful build on circle CI " + }, + { + "type": "link", + "url": "https://app.circleci.com/pipelines/github/OpenLineage/OpenLineage/8271/workflows/0ec521ae-cd21-444a-bfec-554d101770ea" + } + ] + } + ] + } + ], + "team": "T01CWUYP5AR", + "thread_ts": "1697137714.503349", + "parent_user_id": "U05T8BJD4DU" + }, + { + "client_msg_id": "de82063b-d85b-4d99-9675-3d73375d9fd9", + "type": "message", + "text": "... 111 more\nCaused by: java.lang.ClassNotFoundException: org.gradle.api.provider.HasMultipleValues\n ... 117 more", + "user": "U05T8BJD4DU", + "ts": "1697443984.091359", + "blocks": [ + { + "type": "rich_text", + "block_id": "yNzSD", + "elements": [ + { + "type": "rich_text_section", + "elements": [ + { + "type": "text", + "text": "... 111 more\nCaused by: java.lang.ClassNotFoundException: org.gradle.api.provider.HasMultipleValues\n ... 117 more" + } + ] + } + ] + } + ], + "team": "T01CWUYP5AR", + "thread_ts": "1697137714.503349", + "parent_user_id": "U05T8BJD4DU" + }, + { + "client_msg_id": "77981fd1-5be7-4a24-aedd-a6d8a94c4c09", + "type": "message", + "text": "<@U02MK6YNAQ5> now I am doing gradlew instead of gradle on windows coz Linux one doesn't work. The doc didn't mention about setting up Spark / Hadoop and that's my original question -- do I need to setup local Spark? Now it's throwing an error on Hadoop: java.io.FileNotFoundException: java.io.FileNotFoundException: HADOOP_HOME and hadoop.home.dir are unset.", + "user": "U05T8BJD4DU", + "ts": "1697516767.807949", + "blocks": [ + { + "type": "rich_text", + "block_id": "PqJ+j", + "elements": [ + { + "type": "rich_text_section", + "elements": [ + { + "type": "user", + "user_id": "U02MK6YNAQ5" + }, + { + "type": "text", + "text": " now I am doing gradlew instead of gradle on windows coz Linux one doesn't work. The doc didn't mention about setting up Spark / Hadoop and that's my original question -- do I need to setup local Spark? Now it's throwing an error on Hadoop: java.io.FileNotFoundException: java.io.FileNotFoundException: HADOOP_HOME and hadoop.home.dir are unset." + } + ] + } + ] + } + ], + "team": "T01CWUYP5AR", + "thread_ts": "1697137714.503349", + "parent_user_id": "U05T8BJD4DU" + }, + { + "client_msg_id": "5f6c4714-ae33-41d6-ad0b-18efd81e66f9", + "type": "message", + "text": "Got it working with Mac, couldn't get it working with Windows / Linux subsystem", + "user": "U05T8BJD4DU", + "ts": "1697945628.208239", + "blocks": [ + { + "type": "rich_text", + "block_id": "mYifP", + "elements": [ + { + "type": "rich_text_section", + "elements": [ + { + "type": "text", + "text": "Got it working with Mac, couldn't get it working with Windows / Linux subsystem" + } + ] + } + ] + } + ], + "team": "T01CWUYP5AR", + "thread_ts": "1697137714.503349", + "parent_user_id": "U05T8BJD4DU" + }, + { + "client_msg_id": "2cabcbc1-35a9-4380-91b5-8d4020b25075", + "type": "message", + "text": "Now getting class not found despite build and test succeeded", + "user": "U05T8BJD4DU", + "ts": "1697994520.647229", + "blocks": [ + { + "type": "rich_text", + "block_id": "Mceh7", + "elements": [ + { + "type": "rich_text_section", + "elements": [ + { + "type": "text", + "text": "Now getting class not found despite build and test succeeded" + } + ] + } + ] + } + ], + "team": "T01CWUYP5AR", + "thread_ts": "1697137714.503349", + "parent_user_id": "U05T8BJD4DU" + }, + { + "client_msg_id": "ee3c9260-7865-4e88-a230-c398e578ecc6", + "type": "message", + "text": "I uploaded the wrong jar.. there are so many jars, only the jar in the spark folder works, not subfolder", + "user": "U05T8BJD4DU", + "ts": "1698025583.327319", + "blocks": [ + { + "type": "rich_text", + "block_id": "Ix2VZ", + "elements": [ + { + "type": "rich_text_section", + "elements": [ + { + "type": "text", + "text": "I uploaded the wrong jar.. there are so many jars, only the jar in the spark folder works, not subfolder" + } + ] + } + ] + } + ], + "team": "T01CWUYP5AR", + "thread_ts": "1697137714.503349", + "parent_user_id": "U05T8BJD4DU" + } + ] + }, + { + "client_msg_id": "340c47c3-a98d-48d8-a43d-2a7cda30c1e0", + "type": "message", + "text": "\nFriendly reminder: this month’s TSC meeting, open to all, is tomorrow at 10 am PT: ", + "user": "U02LXF3HUN7", + "ts": "1697043601.182719", + "blocks": [ + { + "type": "rich_text", + "block_id": "p7fLH", + "elements": [ + { + "type": "rich_text_section", + "elements": [ + { + "type": "broadcast", + "range": "channel" + }, + { + "type": "text", + "text": "\nFriendly reminder: this month’s TSC meeting, open to all, is tomorrow at 10 am PT: " + }, + { + "type": "link", + "url": "https://openlineage.slack.com/archives/C01CK9T7HKR/p1696531454431629" + } + ] + } + ] + } + ], + "team": "T01CWUYP5AR", + "attachments": [ + { + "from_url": "https://openlineage.slack.com/archives/C01CK9T7HKR/p1696531454431629", + "ts": "1696531454.431629", + "author_id": "U02LXF3HUN7", + "channel_id": "C01CK9T7HKR", + "channel_team": "T01CWUYP5AR", + "is_msg_unfurl": true, + "message_blocks": [ + { + "team": "T01CWUYP5AR", + "channel": "C01CK9T7HKR", + "ts": "1696531454.431629", + "message": { + "blocks": [ + { + "type": "rich_text", + "block_id": "TBGz6", + "elements": [ + { + "type": "rich_text_section", + "elements": [ + { + "type": "broadcast", + "range": "channel", + "style": { + "bold": true + } + }, + { + "type": "text", + "text": "\nThis month’s TSC meeting is next Thursday the 12th at 10am PT. On the tentative agenda:\n" + } + ] + }, + { + "type": "rich_text_list", + "elements": [ + { + "type": "rich_text_section", + "elements": [ + { + "type": "text", + "text": "announcements" + } + ] + }, + { + "type": "rich_text_section", + "elements": [ + { + "type": "text", + "text": "recent releases" + } + ] + }, + { + "type": "rich_text_section", + "elements": [ + { + "type": "text", + "text": "Airflow Summit recap" + } + ] + }, + { + "type": "rich_text_section", + "elements": [ + { + "type": "text", + "text": "tutorial: migrating to the Airflow Provider" + } + ] + }, + { + "type": "rich_text_section", + "elements": [ + { + "type": "text", + "text": "discussion topic: observability for OpenLineage/Marquez" + } + ] + }, + { + "type": "rich_text_section", + "elements": [ + { + "type": "text", + "text": "open discussion" + } + ] + }, + { + "type": "rich_text_section", + "elements": [ + { + "type": "text", + "text": "more (TBA)" + } + ] + } + ], + "style": "bullet", + "indent": 0, + "border": 0 + }, + { + "type": "rich_text_section", + "elements": [ + { + "type": "text", + "text": "More info and the meeting link can be found on the " + }, + { + "type": "link", + "url": "https://openlineage.io/meetings/", + "text": "website" + }, + { + "type": "text", + "text": ". All are welcome! Do you have a discussion topic, use case or integration you’d like to demo? DM me to be added to the agenda." + } + ] + } + ] + } + ] + } + } + ], + "id": 1, + "original_url": "https://openlineage.slack.com/archives/C01CK9T7HKR/p1696531454431629", + "fallback": "[October 5th, 2023 11:44 AM] michael282: **\nThis month’s TSC meeting is next Thursday the 12th at 10am PT. On the tentative agenda:\n• announcements\n• recent releases\n• Airflow Summit recap\n• tutorial: migrating to the Airflow Provider\n• discussion topic: observability for OpenLineage/Marquez\n• open discussion\n• more (TBA)\nMore info and the meeting link can be found on the . All are welcome! Do you have a discussion topic, use case or integration you’d like to demo? DM me to be added to the agenda.", + "text": "**\nThis month’s TSC meeting is next Thursday the 12th at 10am PT. On the tentative agenda:\n• announcements\n• recent releases\n• Airflow Summit recap\n• tutorial: migrating to the Airflow Provider\n• discussion topic: observability for OpenLineage/Marquez\n• open discussion\n• more (TBA)\nMore info and the meeting link can be found on the . All are welcome! Do you have a discussion topic, use case or integration you’d like to demo? DM me to be added to the agenda.", + "author_name": "Michael Robinson", + "author_link": "https://openlineage.slack.com/team/U02LXF3HUN7", + "author_icon": "https://avatars.slack-edge.com/2022-01-25/3019716733729_66fea720e9504dc08144_48.jpg", + "author_subname": "Michael Robinson", + "mrkdwn_in": [ + "text" + ], + "footer": "Slack Conversation" + } + ], + "thread_ts": "1697043601.182719", + "reply_count": 1, + "reply_users_count": 1, + "latest_reply": "1697048805.134799", + "reply_users": [ + "U02LXF3HUN7" + ], + "is_locked": false, + "subscribed": true, + "last_read": "1697048805.134799", + "replies": [ + { + "client_msg_id": "dd879f8b-63c3-4dd5-9414-fe93655d970e", + "type": "message", + "text": "Newly added discussion topics:\n• a proposal to add a Registry of Consumers and Producers\n• a dbt issue to add OpenLineage Dataset names to the Manifest\n• a proposal to add Dataset support in Spark LogicalPlan Nodes\n• a proposal to institute a certification process for new integrations", + "user": "U02LXF3HUN7", + "ts": "1697048805.134799", + "blocks": [ + { + "type": "rich_text", + "block_id": "L1v/X", + "elements": [ + { + "type": "rich_text_section", + "elements": [ + { + "type": "text", + "text": "Newly added discussion topics:\n" + } + ] + }, + { + "type": "rich_text_list", + "elements": [ + { + "type": "rich_text_section", + "elements": [ + { + "type": "text", + "text": "a proposal to add a Registry of Consumers and Producers" + } + ] + }, + { + "type": "rich_text_section", + "elements": [ + { + "type": "text", + "text": "a dbt issue to add OpenLineage Dataset names to the Manifest" + } + ] + }, + { + "type": "rich_text_section", + "elements": [ + { + "type": "text", + "text": "a proposal to add Dataset support in Spark LogicalPlan Nodes" + } + ] + }, + { + "type": "rich_text_section", + "elements": [ + { + "type": "text", + "text": "a proposal to institute a certification process for new integrations" + } + ] + } + ], + "style": "bullet", + "indent": 0, + "border": 0 + } + ] + } + ], + "team": "T01CWUYP5AR", + "edited": { + "user": "U02LXF3HUN7", + "ts": "1697048825.000000" + }, + "thread_ts": "1697043601.182719", + "parent_user_id": "U02LXF3HUN7" + } + ] + }, + { + "client_msg_id": "385c50e2-1da3-406e-a122-600f2da833c4", + "type": "message", + "text": "Hi @there, I am trying to make API call to get column-lineage information could you please let me know the url construct to retrieve the same? As per the API documentation I am passing the following url to GET column-lineage: but getting error code:400. Thanks", + "user": "U05HK41VCH1", + "ts": "1697040264.029839", + "blocks": [ + { + "type": "rich_text", + "block_id": "STdaF", + "elements": [ + { + "type": "rich_text_section", + "elements": [ + { + "type": "text", + "text": "Hi @there, I am trying to make API call to get column-lineage information could you please let me know the url construct to retrieve the same? As per the API documentation I am passing the following url to GET column-lineage: " + }, + { + "type": "link", + "url": "http://localhost:5000/api/v1/column-lineage" + }, + { + "type": "text", + "text": " but getting error code:400. Thanks" + } + ] + } + ] + } + ], + "team": "T01CWUYP5AR", + "thread_ts": "1697040264.029839", + "reply_count": 3, + "reply_users_count": 2, + "latest_reply": "1697537226.424999", + "reply_users": [ + "U01DCMDFHBK", + "U05HK41VCH1" + ], + "is_locked": false, + "subscribed": false, + "replies": [ + { + "client_msg_id": "b64cbd07-3aee-41dd-918a-85279907f1ca", + "type": "message", + "text": "Make sure to provide a dataset field `nodeId` as a query param in your request. If you’ve seeded Marquez with test metadata, you can use:\n```curl -XGET \"\"```\nYou can view the API docs for column lineage !", + "user": "U01DCMDFHBK", + "ts": "1697133326.602989", + "blocks": [ + { + "type": "rich_text", + "block_id": "+yo0+", + "elements": [ + { + "type": "rich_text_section", + "elements": [ + { + "type": "text", + "text": "Make sure to provide a dataset field " + }, + { + "type": "text", + "text": "nodeId", + "style": { + "code": true + } + }, + { + "type": "text", + "text": " as a query param in your request. If you’ve seeded Marquez with test metadata, you can use:\n" + } + ] + }, + { + "type": "rich_text_preformatted", + "elements": [ + { + "type": "text", + "text": "curl -XGET \"" + }, + { + "type": "link", + "url": "http://localhost:5002/api/v1/column-lineage?nodeId=datasetField%3Afood_delivery%3Apublic.delivery_7_days%3Acustomer_email" + }, + { + "type": "text", + "text": "\"" + } + ], + "border": 0 + }, + { + "type": "rich_text_section", + "elements": [ + { + "type": "text", + "text": "You can view the API docs for column lineage " + }, + { + "type": "link", + "url": "https://marquezproject.github.io/marquez/openapi.html#operation/getLineage", + "text": "here" + }, + { + "type": "text", + "text": "!" + } + ] + } + ] + } + ], + "team": "T01CWUYP5AR", + "thread_ts": "1697040264.029839", + "parent_user_id": "U05HK41VCH1" + }, + { + "client_msg_id": "f371513d-c971-4254-8b86-1b79fed2fdd5", + "type": "message", + "text": "Thanks Willy. The documentation says 'name space' so i constructed API Like this:\n''\nbut it is still not working :disappointed:", + "user": "U05HK41VCH1", + "ts": "1697536656.423129", + "blocks": [ + { + "type": "rich_text", + "block_id": "P7W1y", + "elements": [ + { + "type": "rich_text_section", + "elements": [ + { + "type": "text", + "text": "Thanks Willy. The documentation says 'name space' so i constructed API Like this:\n'" + }, + { + "type": "link", + "url": "http://marquez-web:3000/api/v1/column-lineage/nodeId=datasetField:file:/home/jovyan/Downloads/event_attribute.csv:eventType" + }, + { + "type": "text", + "text": "'\nbut it is still not working " + }, + { + "type": "emoji", + "name": "disappointed", + "unicode": "1f61e" + } + ] + } + ] + } + ], + "team": "T01CWUYP5AR", + "thread_ts": "1697040264.029839", + "parent_user_id": "U05HK41VCH1" + }, + { + "client_msg_id": "01407b70-dab5-46ea-a935-cf1fe36d81a9", + "type": "message", + "text": "nodeId is constructed like this: datasetField:<namespace>:<dataset>:<field name>", + "user": "U05HK41VCH1", + "ts": "1697537226.424999", + "blocks": [ + { + "type": "rich_text", + "block_id": "5aVHi", + "elements": [ + { + "type": "rich_text_section", + "elements": [ + { + "type": "text", + "text": "nodeId is constructed like this: datasetField:::" + } + ] + } + ] + } + ], + "team": "T01CWUYP5AR", + "thread_ts": "1697040264.029839", + "parent_user_id": "U05HK41VCH1" + } + ] + }, + { + "client_msg_id": "35fe2423-f25c-4cda-952b-d4cad50f40ba", + "type": "message", + "text": " When i am running this sql as part of a databricks notebook, i am recieving an OL event where i see only an output dataset and there is no input dataset or a symlink facet inside the dataset to map it to the underlying azure storage object. Can anyone kindly help on this\n```spark.sql(f\"CREATE TABLE IF NOT EXISTS covid_research.uscoviddata USING delta LOCATION ''\")\n{\n \"eventTime\": \"2023-10-11T10:47:36.296Z\",\n \"producer\": \"\",\n \"schemaURL\": \"\",\n \"eventType\": \"COMPLETE\",\n \"run\": {\n \"runId\": \"d0f40be9-b921-4c84-ac9f-f14a86c29ff7\",\n \"facets\": {\n \"spark.logicalPlan\": {\n \"_producer\": \"\",\n \"_schemaURL\": \"\",\n \"plan\": [\n {\n \"class\": \"org.apache.spark.sql.catalyst.plans.logical.CreateTable\",\n \"num-children\": 1,\n \"name\": 0,\n \"tableSchema\": [],\n \"partitioning\": [],\n \"tableSpec\": null,\n \"ignoreIfExists\": true\n },\n {\n \"class\": \"org.apache.spark.sql.catalyst.analysis.ResolvedIdentifier\",\n \"num-children\": 0,\n \"catalog\": null,\n \"identifier\": null\n }\n ]\n },\n \"spark_version\": {\n \"_producer\": \"\",\n \"_schemaURL\": \"\",\n \"spark-version\": \"3.3.0\",\n \"openlineage-spark-version\": \"1.2.2\"\n },\n \"processing_engine\": {\n \"_producer\": \"\",\n \"_schemaURL\": \"\",\n \"version\": \"3.3.0\",\n \"name\": \"spark\",\n \"openlineageAdapterVersion\": \"1.2.2\"\n }\n }\n },\n \"job\": {\n \"namespace\": \"default\",\n \"name\": \"adb-3942203504488904.4.azuredatabricks.net.create_table.covid_research_db_uscoviddata\",\n \"facets\": {}\n },\n \"inputs\": [],\n \"outputs\": [\n {\n \"namespace\": \"dbfs\",\n \"name\": \"/user/hive/warehouse/covid_research.db/uscoviddata\",\n \"facets\": {\n \"dataSource\": {\n \"_producer\": \"\",\n \"_schemaURL\": \"\",\n \"name\": \"dbfs\",\n \"uri\": \"dbfs\"\n },\n \"schema\": {\n \"_producer\": \"\",\n \"_schemaURL\": \"\",\n \"fields\": []\n },\n \"storage\": {\n \"_producer\": \"\",\n \"_schemaURL\": \"\",\n \"storageLayer\": \"unity\",\n \"fileFormat\": \"parquet\"\n },\n \"symlinks\": {\n \"_producer\": \"\",\n \"_schemaURL\": \"\",\n \"identifiers\": [\n {\n \"namespace\": \"/user/hive/warehouse/covid_research.db\",\n \"name\": \"covid_research.uscoviddata\",\n \"type\": \"TABLE\"\n }\n ]\n },\n \"lifecycleStateChange\": {\n \"_producer\": \"\",\n \"_schemaURL\": \"\",\n \"lifecycleStateChange\": \"CREATE\"\n }\n },\n \"outputFacets\": {}\n }\n ]\n}```", + "user": "U05QL7LN2GH", + "ts": "1697021758.073929", + "blocks": [ + { + "type": "rich_text", + "block_id": "AFoNq", + "elements": [ + { + "type": "rich_text_section", + "elements": [ + { + "type": "broadcast", + "range": "here" + }, + { + "type": "text", + "text": " When i am running this sql as part of a databricks notebook, i am recieving an OL event where i see only an output dataset and there is no input dataset or a symlink facet inside the dataset to map it to the underlying azure storage object. Can anyone kindly help on this\n" + } + ] + }, + { + "type": "rich_text_preformatted", + "elements": [ + { + "type": "text", + "text": "spark.sql(f\"CREATE TABLE IF NOT EXISTS covid_research.uscoviddata USING delta LOCATION '" + }, + { + "type": "link", + "url": "abfss://oltptestdata@jeevanacceldata.dfs.core.windows.net/testdata/modified-delta" + }, + { + "type": "text", + "text": "'\")\n{\n \"eventTime\": \"2023-10-11T10:47:36.296Z\",\n \"producer\": \"" + }, + { + "type": "link", + "url": "https://github.com/OpenLineage/OpenLineage/tree/1.2.2/integration/spark" + }, + { + "type": "text", + "text": "\",\n \"schemaURL\": \"" + }, + { + "type": "link", + "url": "https://openlineage.io/spec/2-0-2/OpenLineage.json#/$defs/RunEvent" + }, + { + "type": "text", + "text": "\",\n \"eventType\": \"COMPLETE\",\n \"run\": {\n \"runId\": \"d0f40be9-b921-4c84-ac9f-f14a86c29ff7\",\n \"facets\": {\n \"spark.logicalPlan\": {\n \"_producer\": \"" + }, + { + "type": "link", + "url": "https://github.com/OpenLineage/OpenLineage/tree/1.2.2/integration/spark" + }, + { + "type": "text", + "text": "\",\n \"_schemaURL\": \"" + }, + { + "type": "link", + "url": "https://openlineage.io/spec/2-0-2/OpenLineage.json#/$defs/RunFacet" + }, + { + "type": "text", + "text": "\",\n \"plan\": [\n {\n \"class\": \"org.apache.spark.sql.catalyst.plans.logical.CreateTable\",\n \"num-children\": 1,\n \"name\": 0,\n \"tableSchema\": [],\n \"partitioning\": [],\n \"tableSpec\": null,\n \"ignoreIfExists\": true\n },\n {\n \"class\": \"org.apache.spark.sql.catalyst.analysis.ResolvedIdentifier\",\n \"num-children\": 0,\n \"catalog\": null,\n \"identifier\": null\n }\n ]\n },\n \"spark_version\": {\n \"_producer\": \"" + }, + { + "type": "link", + "url": "https://github.com/OpenLineage/OpenLineage/tree/1.2.2/integration/spark" + }, + { + "type": "text", + "text": "\",\n \"_schemaURL\": \"" + }, + { + "type": "link", + "url": "https://openlineage.io/spec/2-0-2/OpenLineage.json#/$defs/RunFacet" + }, + { + "type": "text", + "text": "\",\n \"spark-version\": \"3.3.0\",\n \"openlineage-spark-version\": \"1.2.2\"\n },\n \"processing_engine\": {\n \"_producer\": \"" + }, + { + "type": "link", + "url": "https://github.com/OpenLineage/OpenLineage/tree/1.2.2/integration/spark" + }, + { + "type": "text", + "text": "\",\n \"_schemaURL\": \"" + }, + { + "type": "link", + "url": "https://openlineage.io/spec/facets/1-1-0/ProcessingEngineRunFacet.json#/$defs/ProcessingEngineRunFacet" + }, + { + "type": "text", + "text": "\",\n \"version\": \"3.3.0\",\n \"name\": \"spark\",\n \"openlineageAdapterVersion\": \"1.2.2\"\n }\n }\n },\n \"job\": {\n \"namespace\": \"default\",\n \"name\": \"adb-3942203504488904.4.azuredatabricks.net.create_table.covid_research_db_uscoviddata\",\n \"facets\": {}\n },\n \"inputs\": [],\n \"outputs\": [\n {\n \"namespace\": \"dbfs\",\n \"name\": \"/user/hive/warehouse/covid_research.db/uscoviddata\",\n \"facets\": {\n \"dataSource\": {\n \"_producer\": \"" + }, + { + "type": "link", + "url": "https://github.com/OpenLineage/OpenLineage/tree/1.2.2/integration/spark" + }, + { + "type": "text", + "text": "\",\n \"_schemaURL\": \"" + }, + { + "type": "link", + "url": "https://openlineage.io/spec/facets/1-0-0/DatasourceDatasetFacet.json#/$defs/DatasourceDatasetFacet" + }, + { + "type": "text", + "text": "\",\n \"name\": \"dbfs\",\n \"uri\": \"dbfs\"\n },\n \"schema\": {\n \"_producer\": \"" + }, + { + "type": "link", + "url": "https://github.com/OpenLineage/OpenLineage/tree/1.2.2/integration/spark" + }, + { + "type": "text", + "text": "\",\n \"_schemaURL\": \"" + }, + { + "type": "link", + "url": "https://openlineage.io/spec/facets/1-0-0/SchemaDatasetFacet.json#/$defs/SchemaDatasetFacet" + }, + { + "type": "text", + "text": "\",\n \"fields\": []\n },\n \"storage\": {\n \"_producer\": \"" + }, + { + "type": "link", + "url": "https://github.com/OpenLineage/OpenLineage/tree/1.2.2/integration/spark" + }, + { + "type": "text", + "text": "\",\n \"_schemaURL\": \"" + }, + { + "type": "link", + "url": "https://openlineage.io/spec/facets/1-0-0/StorageDatasetFacet.json#/$defs/StorageDatasetFacet" + }, + { + "type": "text", + "text": "\",\n \"storageLayer\": \"unity\",\n \"fileFormat\": \"parquet\"\n },\n \"symlinks\": {\n \"_producer\": \"" + }, + { + "type": "link", + "url": "https://github.com/OpenLineage/OpenLineage/tree/1.2.2/integration/spark" + }, + { + "type": "text", + "text": "\",\n \"_schemaURL\": \"" + }, + { + "type": "link", + "url": "https://openlineage.io/spec/facets/1-0-0/SymlinksDatasetFacet.json#/$defs/SymlinksDatasetFacet" + }, + { + "type": "text", + "text": "\",\n \"identifiers\": [\n {\n \"namespace\": \"/user/hive/warehouse/covid_research.db\",\n \"name\": \"covid_research.uscoviddata\",\n \"type\": \"TABLE\"\n }\n ]\n },\n \"lifecycleStateChange\": {\n \"_producer\": \"" + }, + { + "type": "link", + "url": "https://github.com/OpenLineage/OpenLineage/tree/1.2.2/integration/spark" + }, + { + "type": "text", + "text": "\",\n \"_schemaURL\": \"" + }, + { + "type": "link", + "url": "https://openlineage.io/spec/facets/1-0-0/LifecycleStateChangeDatasetFacet.json#/$defs/LifecycleStateChangeDatasetFacet" + }, + { + "type": "text", + "text": "\",\n \"lifecycleStateChange\": \"CREATE\"\n }\n },\n \"outputFacets\": {}\n }\n ]\n}" + } + ], + "border": 0 + } + ] + } + ], + "team": "T01CWUYP5AR", + "thread_ts": "1697021758.073929", + "reply_count": 3, + "reply_users_count": 2, + "latest_reply": "1697021974.938879", + "reply_users": [ + "U05FLJE4GDU", + "U05QL7LN2GH" + ], + "is_locked": false, + "subscribed": false, + "replies": [ + { + "client_msg_id": "db148454-967c-4e72-942e-54a088b71cdf", + "type": "message", + "text": "Hey Guntaka - can I ask you a favour? Can you please stop using `@here` or `@channel` - please keep in mind, you're pinging over 1000 people when you use that mention. Its incredibly distracting to have Slack notify me of a message that isn't pertinent to me.", + "user": "U05FLJE4GDU", + "ts": "1697021866.183179", + "blocks": [ + { + "type": "rich_text", + "block_id": "HKa8J", + "elements": [ + { + "type": "rich_text_section", + "elements": [ + { + "type": "text", + "text": "Hey Guntaka - can I ask you a favour? Can you please stop using " + }, + { + "type": "text", + "text": "@here", + "style": { + "code": true + } + }, + { + "type": "text", + "text": " or " + }, + { + "type": "text", + "text": "@channel", + "style": { + "code": true + } + }, + { + "type": "text", + "text": " - please keep in mind, you're pinging over 1000 people when you use that mention. Its incredibly distracting to have Slack notify me of a message that isn't pertinent to me." + } + ] + } + ] + } + ], + "team": "T01CWUYP5AR", + "thread_ts": "1697021758.073929", + "parent_user_id": "U05QL7LN2GH" + }, + { + "client_msg_id": "7357be3d-f271-4190-9a03-aaefe7e602cf", + "type": "message", + "text": "sure noted <@U05FLJE4GDU>", + "user": "U05QL7LN2GH", + "ts": "1697021930.666629", + "blocks": [ + { + "type": "rich_text", + "block_id": "KBawo", + "elements": [ + { + "type": "rich_text_section", + "elements": [ + { + "type": "text", + "text": "sure noted " + }, + { + "type": "user", + "user_id": "U05FLJE4GDU" + } + ] + } + ] + } + ], + "team": "T01CWUYP5AR", + "thread_ts": "1697021758.073929", + "parent_user_id": "U05QL7LN2GH" + }, + { + "client_msg_id": "6180e304-09ba-4165-b9c5-a2e6ddf55398", + "type": "message", + "text": "Thank you!", + "user": "U05FLJE4GDU", + "ts": "1697021974.938879", + "blocks": [ + { + "type": "rich_text", + "block_id": "Ye0AT", + "elements": [ + { + "type": "rich_text_section", + "elements": [ + { + "type": "text", + "text": "Thank you!" + } + ] + } + ] + } + ], + "team": "T01CWUYP5AR", + "thread_ts": "1697021758.073929", + "parent_user_id": "U05QL7LN2GH" + } + ] + }, + { + "client_msg_id": "73ca512c-b9c3-4e01-ac03-f7c97c65b991", + "type": "message", + "text": " i am trying out the databricks spark integration and in one of the events i am getting a openlineage event where the output dataset is having a facet called `symlinks` , the statement that generated this event is this sql\n```CREATE TABLE IF NOT EXISTS covid_research.covid_data \nUSING CSV\nLOCATION '' \nOPTIONS (header \"true\", inferSchema \"true\");```\nCan someone kindly let me know what this `symlinks` facet is. i tried seeing the spec but did not get it completely", + "user": "U05QL7LN2GH", + "ts": "1696995819.546399", + "blocks": [ + { + "type": "rich_text", + "block_id": "N0r6R", + "elements": [ + { + "type": "rich_text_section", + "elements": [ + { + "type": "broadcast", + "range": "here" + }, + { + "type": "text", + "text": " i am trying out the databricks spark integration and in one of the events i am getting a openlineage event where the output dataset is having a facet called " + }, + { + "type": "text", + "text": "symlinks", + "style": { + "code": true + } + }, + { + "type": "text", + "text": " , the statement that generated this event is this sql\n" + } + ] + }, + { + "type": "rich_text_preformatted", + "elements": [ + { + "type": "text", + "text": "CREATE TABLE IF NOT EXISTS covid_research.covid_data \nUSING CSV\nLOCATION '" + }, + { + "type": "link", + "url": "abfss://oltptestdata@jeevanacceldata.dfs.core.windows.net/testdata/johns-hopkins-covid-19-daily-dashboard-cases-by-states.csv" + }, + { + "type": "text", + "text": "' \nOPTIONS (header \"true\", inferSchema \"true\");" + } + ], + "border": 0 + }, + { + "type": "rich_text_section", + "elements": [ + { + "type": "text", + "text": "Can someone kindly let me know what this " + }, + { + "type": "text", + "text": "symlinks", + "style": { + "code": true + } + }, + { + "type": "text", + "text": " facet is. i tried seeing the spec but did not get it completely" + } + ] + } + ] + } + ], + "team": "T01CWUYP5AR", + "thread_ts": "1696995819.546399", + "reply_count": 3, + "reply_users_count": 2, + "latest_reply": "1697001944.554579", + "reply_users": [ + "U05T8BJD4DU", + "U05QL7LN2GH" + ], + "is_locked": false, + "subscribed": false, + "replies": [ + { + "client_msg_id": "a2c46736-b265-4087-a5b4-900d15a2dfab", + "type": "message", + "text": "I use it to get the table with database name", + "user": "U05T8BJD4DU", + "ts": "1696995893.445939", + "blocks": [ + { + "type": "rich_text", + "block_id": "Hdg1w", + "elements": [ + { + "type": "rich_text_section", + "elements": [ + { + "type": "text", + "text": "I use it to get the table with database name" + } + ] + } + ] + } + ], + "team": "T01CWUYP5AR", + "thread_ts": "1696995819.546399", + "parent_user_id": "U05QL7LN2GH" + }, + { + "client_msg_id": "e7c1d108-3df6-49fd-8cdc-70381e9b03bb", + "type": "message", + "text": "so can i think it like if there is a synlink, then that table is kind of a reference to the original dataset", + "user": "U05QL7LN2GH", + "ts": "1696996035.076329", + "blocks": [ + { + "type": "rich_text", + "block_id": "D+R+W", + "elements": [ + { + "type": "rich_text_section", + "elements": [ + { + "type": "text", + "text": "so can i think it like if there is a synlink, then that table is kind of a reference to the original dataset" + } + ] + } + ] + } + ], + "team": "T01CWUYP5AR", + "thread_ts": "1696995819.546399", + "parent_user_id": "U05QL7LN2GH" + }, + { + "client_msg_id": "514dff8b-7ce4-4a68-9d4c-e2eb623a1a1f", + "type": "message", + "text": "yes", + "user": "U05T8BJD4DU", + "ts": "1697001944.554579", + "blocks": [ + { + "type": "rich_text", + "block_id": "IB8ze", + "elements": [ + { + "type": "rich_text_section", + "elements": [ + { + "type": "text", + "text": "yes" + } + ] + } + ] + } + ], + "team": "T01CWUYP5AR", + "thread_ts": "1696995819.546399", + "parent_user_id": "U05QL7LN2GH", + "reactions": [ + { + "name": "raised_hands", + "users": [ + "U02MK6YNAQ5" + ], + "count": 1 + } + ] + } + ] + }, + { + "client_msg_id": "98148df4-23e3-4211-806d-5954410eabfb", + "type": "message", + "text": "example:\n\n```{\"environment-properties\":{\"spark.databricks.clusterUsageTags.clusterName\":\"'s Cluster\",\"spark.databricks.job.runId\":\"\",\"spark.databricks.job.type\":\"\",\"spark.databricks.clusterUsageTags.azureSubscriptionId\":\"a4f54399-8db8-4849-adcc-a42aed1fb97f\",\"spark.databricks.notebook.path\":\"/Repos/jason.yip@tredence.com/segmentation/01_Data Prep\",\"spark.databricks.clusterUsageTags.clusterOwnerOrgId\":\"4679476628690204\",\"MountPoints\":[{\"MountPoint\":\"/databricks-datasets\",\"Source\":\"databricks-datasets\"},{\"MountPoint\":\"/Volumes\",\"Source\":\"UnityCatalogVolumes\"},{\"MountPoint\":\"/databricks/mlflow-tracking\",\"Source\":\"databricks/mlflow-tracking\"},{\"MountPoint\":\"/databricks-results\",\"Source\":\"databricks-results\"},{\"MountPoint\":\"/databricks/mlflow-registry\",\"Source\":\"databricks/mlflow-registry\"},{\"MountPoint\":\"/Volume\",\"Source\":\"DbfsReserved\"},{\"MountPoint\":\"/volumes\",\"Source\":\"DbfsReserved\"},{\"MountPoint\":\"/\",\"Source\":\"DatabricksRoot\"},{\"MountPoint\":\"/volume\",\"Source\":\"DbfsReserved\"}],\"User\":\"\",\"UserId\":\"4768657035718622\",\"OrgId\":\"4679476628690204\"}}```", + "user": "U05T8BJD4DU", + "ts": "1696985639.868119", + "blocks": [ + { + "type": "rich_text", + "block_id": "Av2WJ", + "elements": [ + { + "type": "rich_text_section", + "elements": [ + { + "type": "text", + "text": "example:\n\n" + } + ] + }, + { + "type": "rich_text_preformatted", + "elements": [ + { + "type": "text", + "text": "{\"environment-properties\":{\"spark.databricks.clusterUsageTags.clusterName\":\"jason.yip@tredence.com's Cluster\",\"spark.databricks.job.runId\":\"\",\"spark.databricks.job.type\":\"\",\"spark.databricks.clusterUsageTags.azureSubscriptionId\":\"a4f54399-8db8-4849-adcc-a42aed1fb97f\",\"spark.databricks.notebook.path\":\"/Repos/jason.yip@tredence.com/segmentation/01_Data Prep\",\"spark.databricks.clusterUsageTags.clusterOwnerOrgId\":\"4679476628690204\",\"MountPoints\":[{\"MountPoint\":\"/databricks-datasets\",\"Source\":\"databricks-datasets\"},{\"MountPoint\":\"/Volumes\",\"Source\":\"UnityCatalogVolumes\"},{\"MountPoint\":\"/databricks/mlflow-tracking\",\"Source\":\"databricks/mlflow-tracking\"},{\"MountPoint\":\"/databricks-results\",\"Source\":\"databricks-results\"},{\"MountPoint\":\"/databricks/mlflow-registry\",\"Source\":\"databricks/mlflow-registry\"},{\"MountPoint\":\"/Volume\",\"Source\":\"DbfsReserved\"},{\"MountPoint\":\"/volumes\",\"Source\":\"DbfsReserved\"},{\"MountPoint\":\"/\",\"Source\":\"DatabricksRoot\"},{\"MountPoint\":\"/volume\",\"Source\":\"DbfsReserved\"}],\"User\":\"jason.yip@tredence.com\",\"UserId\":\"4768657035718622\",\"OrgId\":\"4679476628690204\"}}" + } + ], + "border": 0 + } + ] + } + ], + "team": "T01CWUYP5AR", + "thread_ts": "1696985639.868119", + "reply_count": 4, + "reply_users_count": 2, + "latest_reply": "1697439536.801279", + "reply_users": [ + "U02MK6YNAQ5", + "U05T8BJD4DU" + ], + "is_locked": false, + "subscribed": false, + "replies": [ + { + "client_msg_id": "dd99f2a1-192d-453b-8e43-7712cea81387", + "type": "message", + "text": "Is this related to any OL version? In OL 1.2.2. we've added extra variable ``spark.databricks.clusterUsageTags.clusterAllTags`` to be captured, but this should not break things.\n\nI think we're facing some issues on recent databricks runtime versions. Here is an issue for this: \n\nIs the problem you describe specific to some databricks runtime versions?", + "user": "U02MK6YNAQ5", + "ts": "1697010373.970599", + "blocks": [ + { + "type": "rich_text", + "block_id": "07u8a", + "elements": [ + { + "type": "rich_text_section", + "elements": [ + { + "type": "text", + "text": "Is this related to any OL version? In OL 1.2.2. we've added extra variable `" + }, + { + "type": "text", + "text": "spark.databricks.clusterUsageTags.clusterAllTags`", + "style": { + "code": true + } + }, + { + "type": "text", + "text": " to be captured, but this should not break things.\n\nI think we're facing some issues on recent databricks runtime versions. Here is an issue for this: " + }, + { + "type": "link", + "url": "https://github.com/OpenLineage/OpenLineage/issues/2131" + }, + { + "type": "text", + "text": "\n\nIs the problem you describe specific to some databricks runtime versions?" + } + ] + } + ] + } + ], + "team": "T01CWUYP5AR", + "attachments": [ + { + "id": 1, + "footer_icon": "https://slack.github.com/static/img/favicon-neutral.png", + "ts": 1695709470, + "color": "36a64f", + "bot_id": "B01VA0FB340", + "app_unfurl_url": "https://github.com/OpenLineage/OpenLineage/issues/2131", + "is_app_unfurl": true, + "app_id": "A01BP7R4KNY", + "fallback": "#2131 [Spark Databricks] NoSuchMethodError on ReplaceTableAsSelect", + "text": "More details here: \nLooks like different class implementation on databricks platform.", + "title": "#2131 [Spark Databricks] NoSuchMethodError on ReplaceTableAsSelect", + "title_link": "https://github.com/OpenLineage/OpenLineage/issues/2131", + "footer": "", + "fields": [ + { + "value": "integration/spark, integration/databricks", + "title": "Labels", + "short": true + } + ], + "mrkdwn_in": [ + "text" + ] + } + ], + "thread_ts": "1696985639.868119", + "parent_user_id": "U05T8BJD4DU" + }, + { + "client_msg_id": "0c2505bc-3772-41ad-a213-bb6ad7b7cdd3", + "type": "message", + "text": "yes, exactly Spark 3.4+", + "user": "U05T8BJD4DU", + "ts": "1697037426.350799", + "blocks": [ + { + "type": "rich_text", + "block_id": "TXfld", + "elements": [ + { + "type": "rich_text_section", + "elements": [ + { + "type": "text", + "text": "yes, exactly Spark 3.4+" + } + ] + } + ] + } + ], + "team": "T01CWUYP5AR", + "thread_ts": "1696985639.868119", + "parent_user_id": "U05T8BJD4DU" + }, + { + "client_msg_id": "3ae5348a-6474-49f5-8dd8-df7aaf32bb4d", + "type": "message", + "text": "Btw I don't understand the code flow entirely, if we are talking about a different classpath only, I see there's Unity Catalog handler in the code and it says it works the same as Delta, but I am not seeing it subclassing Delta. I suppose it will work the same. \n\nI am happy to jump on a call to show you if needed", + "user": "U05T8BJD4DU", + "ts": "1697073147.278359", + "blocks": [ + { + "type": "rich_text", + "block_id": "31LD4", + "elements": [ + { + "type": "rich_text_section", + "elements": [ + { + "type": "text", + "text": "Btw I don't understand the code flow entirely, if we are talking about a different classpath only, I see there's Unity Catalog handler in the code and it says it works the same as Delta, but I am not seeing it subclassing Delta. I suppose it will work the same. \n\nI am happy to jump on a call to show you if needed" + } + ] + } + ] + } + ], + "team": "T01CWUYP5AR", + "edited": { + "user": "U05T8BJD4DU", + "ts": "1697073159.000000" + }, + "thread_ts": "1696985639.868119", + "parent_user_id": "U05T8BJD4DU" + }, + { + "client_msg_id": "dfa5f011-7ccc-45dd-8ea4-36537c2ff811", + "type": "message", + "text": "<@U02MK6YNAQ5> do you think in Spark 3.4+ only one event would happen?\n\n /**\n * We get exact copies of OL events for org.apache.spark.scheduler.SparkListenerJobStart and\n * org.apache.spark.sql.execution.ui.SparkListenerSQLExecutionStart. The same happens for end\n * events.\n *\n * @return\n */\n private boolean isOnJobStartOrEnd(SparkListenerEvent event) {\n return event instanceof SparkListenerJobStart || event instanceof SparkListenerJobEnd;\n }", + "user": "U05T8BJD4DU", + "ts": "1697439536.801279", + "blocks": [ + { + "type": "rich_text", + "block_id": "Ijzgw", + "elements": [ + { + "type": "rich_text_section", + "elements": [ + { + "type": "user", + "user_id": "U02MK6YNAQ5" + }, + { + "type": "text", + "text": " do you think in Spark 3.4+ only one event would happen?\n\n /**\n * We get exact copies of OL events for org.apache.spark.scheduler.SparkListenerJobStart and\n * org.apache.spark.sql.execution.ui.SparkListenerSQLExecutionStart. The same happens for end\n * events.\n *\n * @return\n */\n private boolean isOnJobStartOrEnd(SparkListenerEvent event) {\n return event instanceof SparkListenerJobStart || event instanceof SparkListenerJobEnd;\n }" + } + ] + } + ] + } + ], + "team": "T01CWUYP5AR", + "thread_ts": "1696985639.868119", + "parent_user_id": "U05T8BJD4DU" + } + ] + }, + { + "client_msg_id": "3e35ff82-64c5-47e0-9c43-e5e2ae88094d", + "type": "message", + "text": "Any idea why \"environment-properties\" is gone in Spark 3.4+ in StartEvent?", + "user": "U05T8BJD4DU", + "ts": "1696914311.793789", + "blocks": [ + { + "type": "rich_text", + "block_id": "4qums", + "elements": [ + { + "type": "rich_text_section", + "elements": [ + { + "type": "text", + "text": "Any idea why \"environment-properties\" is gone in Spark 3.4+ in StartEvent?" + } + ] + } + ] + } + ], + "team": "T01CWUYP5AR" + }, + { + "client_msg_id": "8363d0f5-1aa7-4f2e-9cbd-ce61843beea3", + "type": "message", + "text": "Hello. I am getting started with OL and Marquez with dbt. I am using dbt-ol. The namespace of the dataset showing up in Marquez is not the namespace I provide using OPENLINEAGE_NAMESPACE. It happens to be the same as the source in Marquez which is the snowflake account uri. It's obviously picking up the other env variable OPENLINEAGE_URL so i am pretty sure its not the environment. Is this expected?", + "user": "U021QJMRP47", + "ts": "1696884935.692409", + "blocks": [ + { + "type": "rich_text", + "block_id": "DM7nS", + "elements": [ + { + "type": "rich_text_section", + "elements": [ + { + "type": "text", + "text": "Hello. I am getting started with OL and Marquez with dbt. I am using dbt-ol. The namespace of the dataset showing up in Marquez is not the namespace I provide using OPENLINEAGE_NAMESPACE. It happens to be the same as the source in Marquez which is the snowflake account uri. It's obviously picking up the other env variable OPENLINEAGE_URL so i am pretty sure its not the environment. Is this expected?" + } + ] + } + ] + } + ], + "team": "T01CWUYP5AR", + "thread_ts": "1696884935.692409", + "reply_count": 1, + "reply_users_count": 1, + "latest_reply": "1696892173.535019", + "reply_users": [ + "U02LXF3HUN7" + ], + "is_locked": false, + "subscribed": true, + "last_read": "1696892173.535019", + "replies": [ + { + "client_msg_id": "0fc68344-e20e-4d3f-9948-577766f8043a", + "type": "message", + "text": "Hi Drew, thank you for using OpenLineage! I don’t know the details of your use case, but I believe this is expected, yes. In general, the dataset namespace is different. Jobs are namespaced separately from datasets, which are namespaced by their containing datasources. This is the case so datasets have the same name regardless of the job writing to them, as datasets are sometimes shared by jobs in different namespaces.", + "user": "U02LXF3HUN7", + "ts": "1696892173.535019", + "blocks": [ + { + "type": "rich_text", + "block_id": "h6sQS", + "elements": [ + { + "type": "rich_text_section", + "elements": [ + { + "type": "text", + "text": "Hi Drew, thank you for using OpenLineage! I don’t know the details of your use case, but I believe this is expected, yes. In general, the dataset namespace is different. Jobs are namespaced separately from datasets, which are namespaced by their containing datasources. This is the case so datasets have the same name regardless of the job writing to them, as datasets are sometimes shared by jobs in different namespaces." + } + ] + } + ] + } + ], + "team": "T01CWUYP5AR", + "thread_ts": "1696884935.692409", + "parent_user_id": "U021QJMRP47", + "reactions": [ + { + "name": "+1", + "users": [ + "U021QJMRP47" + ], + "count": 1 + } + ] + } + ] + }, + { + "client_msg_id": "69cca1dd-d1be-466e-babd-e3492cfe5a61", + "type": "message", + "text": "\n*We released OpenLineage 1.4.1!*\n*Additions:*\n• *Client:* *allow setting client’s endpoint via environment variable* <@U01HVNU6A4C> \n• *Flink: expand Iceberg source types* <@U05QA2D1XNV> \n• *Spark: add debug facet* <@U02MK6YNAQ5> \n• *Spark: enable Nessie REST catalog* \nThanks to all the contributors, especially new contributors <@U05QA2D1XNV> and !\n*Release:* \n*Changelog:* \n*Commit history:* \n*Maven:* \n*PyPI:* ", + "user": "U02LXF3HUN7", + "ts": "1696879514.895109", + "blocks": [ + { + "type": "rich_text", + "block_id": "uk4kh", + "elements": [ + { + "type": "rich_text_section", + "elements": [ + { + "type": "broadcast", + "range": "channel" + }, + { + "type": "text", + "text": "\n" + }, + { + "type": "text", + "text": "We released OpenLineage 1.4.1!", + "style": { + "bold": true + } + }, + { + "type": "text", + "text": "\n" + }, + { + "type": "text", + "text": "Additions:", + "style": { + "bold": true + } + }, + { + "type": "text", + "text": "\n" + } + ] + }, + { + "type": "rich_text_list", + "elements": [ + { + "type": "rich_text_section", + "elements": [ + { + "type": "text", + "text": "Client:", + "style": { + "bold": true + } + }, + { + "type": "text", + "text": " " + }, + { + "type": "text", + "text": "allow setting client’s endpoint via environment variable", + "style": { + "bold": true + } + }, + { + "type": "text", + "text": " " + }, + { + "type": "link", + "url": "http://2151.com/OpenLineage/OpenLineage/pull/2151", + "text": "2151" + }, + { + "type": "text", + "text": " " + }, + { + "type": "user", + "user_id": "U01HVNU6A4C" + }, + { + "type": "text", + "text": " " + } + ] + }, + { + "type": "rich_text_section", + "elements": [ + { + "type": "text", + "text": "Flink: expand Iceberg source types", + "style": { + "bold": true + } + }, + { + "type": "text", + "text": " " + }, + { + "type": "link", + "url": "https://github.com/OpenLineage/OpenLineage/pull/2149", + "text": "2149" + }, + { + "type": "text", + "text": " " + }, + { + "type": "user", + "user_id": "U05QA2D1XNV" + }, + { + "type": "text", + "text": " " + } + ] + }, + { + "type": "rich_text_section", + "elements": [ + { + "type": "text", + "text": "Spark: add debug facet", + "style": { + "bold": true + } + }, + { + "type": "text", + "text": " " + }, + { + "type": "link", + "url": "https://github.com/OpenLineage/OpenLineage/pull/2147", + "text": "2147" + }, + { + "type": "text", + "text": " " + }, + { + "type": "user", + "user_id": "U02MK6YNAQ5" + }, + { + "type": "text", + "text": " " + } + ] + }, + { + "type": "rich_text_section", + "elements": [ + { + "type": "text", + "text": "Spark: enable Nessie REST catalog", + "style": { + "bold": true + } + }, + { + "type": "text", + "text": " " + }, + { + "type": "link", + "url": "https://github.com/OpenLineage/OpenLineage/pull/2165", + "text": "2165" + }, + { + "type": "text", + "text": " " + }, + { + "type": "link", + "url": "https://github.com/julwin", + "text": "@julwin", + "unsafe": true + }, + { + "type": "text", + "text": " " + } + ] + } + ], + "style": "bullet", + "indent": 0, + "border": 0 + }, + { + "type": "rich_text_section", + "elements": [ + { + "type": "text", + "text": "Thanks to all the contributors, especially new contributors " + }, + { + "type": "user", + "user_id": "U05QA2D1XNV" + }, + { + "type": "text", + "text": " and " + }, + { + "type": "link", + "url": "https://github.com/julwin", + "text": "@julwin", + "unsafe": true + }, + { + "type": "text", + "text": "!\n" + }, + { + "type": "text", + "text": "Release:", + "style": { + "bold": true + } + }, + { + "type": "text", + "text": " " + }, + { + "type": "link", + "url": "https://github.com/OpenLineage/OpenLineage/releases/tag/1.4.1" + }, + { + "type": "text", + "text": "\n" + }, + { + "type": "text", + "text": "Changelog: ", + "style": { + "bold": true + } + }, + { + "type": "link", + "url": "https://github.com/OpenLineage/OpenLineage/blob/main/CHANGELOG.md" + }, + { + "type": "text", + "text": "\n" + }, + { + "type": "text", + "text": "Commit history:", + "style": { + "bold": true + } + }, + { + "type": "text", + "text": " " + }, + { + "type": "link", + "url": "https://github.com/OpenLineage/OpenLineage/compare/1.3.1...1.4.1" + }, + { + "type": "text", + "text": "\n" + }, + { + "type": "text", + "text": "Maven:", + "style": { + "bold": true + } + }, + { + "type": "text", + "text": " " + }, + { + "type": "link", + "url": "https://oss.sonatype.org/#nexus-search;quick~openlineage" + }, + { + "type": "text", + "text": "\n" + }, + { + "type": "text", + "text": "PyPI:", + "style": { + "bold": true + } + }, + { + "type": "text", + "text": " " + }, + { + "type": "link", + "url": "https://pypi.org/project/openlineage-python/" + } + ] + } + ] + } + ], + "team": "T01CWUYP5AR", + "reactions": [ + { + "name": "+1", + "users": [ + "U05T8BJD4DU", + "U053LLVTHRN", + "U01HVNU6A4C", + "U01HNKK4XAM", + "U05TU0U224A" + ], + "count": 5 + } + ] + }, + { + "type": "message", + "text": " I am trying out the openlineage integration of spark on databricks. There is no event getting emitted from Openlineage, I see logs saying OpenLineage Event Skipped. I am attaching the Notebook that i am trying to run and the cluster logs. Kindly can someone help me on this", + "files": [ + { + "id": "F0608U3FJ3D", + "created": 1696823902, + "timestamp": 1696823902, + "name": "Screenshot 2023-10-09 at 9.28.18 AM.png", + "title": "Screenshot 2023-10-09 at 9.28.18 AM.png", + "mimetype": "image/png", + "filetype": "png", + "pretty_type": "PNG", + "user": "U05QL7LN2GH", + "user_team": "T01CWUYP5AR", + "editable": false, + "size": 375392, + "mode": "hosted", + "is_external": false, + "external_type": "", + "is_public": true, + "public_url_shared": false, + "display_as_bot": false, + "username": "", + "url_private": "https://files.slack.com/files-pri/T01CWUYP5AR-F0608U3FJ3D/screenshot_2023-10-09_at_9.28.18_am.png", + "url_private_download": "https://files.slack.com/files-pri/T01CWUYP5AR-F0608U3FJ3D/download/screenshot_2023-10-09_at_9.28.18_am.png", + "media_display_type": "unknown", + "thumb_64": "https://files.slack.com/files-tmb/T01CWUYP5AR-F0608U3FJ3D-886b1a6404/screenshot_2023-10-09_at_9.28.18_am_64.png", + "thumb_80": "https://files.slack.com/files-tmb/T01CWUYP5AR-F0608U3FJ3D-886b1a6404/screenshot_2023-10-09_at_9.28.18_am_80.png", + "thumb_360": "https://files.slack.com/files-tmb/T01CWUYP5AR-F0608U3FJ3D-886b1a6404/screenshot_2023-10-09_at_9.28.18_am_360.png", + "thumb_360_w": 360, + "thumb_360_h": 178, + "thumb_480": "https://files.slack.com/files-tmb/T01CWUYP5AR-F0608U3FJ3D-886b1a6404/screenshot_2023-10-09_at_9.28.18_am_480.png", + "thumb_480_w": 480, + "thumb_480_h": 238, + "thumb_160": "https://files.slack.com/files-tmb/T01CWUYP5AR-F0608U3FJ3D-886b1a6404/screenshot_2023-10-09_at_9.28.18_am_160.png", + "thumb_720": "https://files.slack.com/files-tmb/T01CWUYP5AR-F0608U3FJ3D-886b1a6404/screenshot_2023-10-09_at_9.28.18_am_720.png", + "thumb_720_w": 720, + "thumb_720_h": 356, + "thumb_800": "https://files.slack.com/files-tmb/T01CWUYP5AR-F0608U3FJ3D-886b1a6404/screenshot_2023-10-09_at_9.28.18_am_800.png", + "thumb_800_w": 800, + "thumb_800_h": 396, + "thumb_960": "https://files.slack.com/files-tmb/T01CWUYP5AR-F0608U3FJ3D-886b1a6404/screenshot_2023-10-09_at_9.28.18_am_960.png", + "thumb_960_w": 960, + "thumb_960_h": 475, + "thumb_1024": "https://files.slack.com/files-tmb/T01CWUYP5AR-F0608U3FJ3D-886b1a6404/screenshot_2023-10-09_at_9.28.18_am_1024.png", + "thumb_1024_w": 1024, + "thumb_1024_h": 507, + "original_w": 2614, + "original_h": 1294, + "thumb_tiny": "AwAXADDRwQelKaWkNADfm/yKXml70UAHNGKKKAFpDQ1B6UAGPeigUtACUY96Wg9KAP/Z", + "permalink": "https://openlineage.slack.com/files/U05QL7LN2GH/F0608U3FJ3D/screenshot_2023-10-09_at_9.28.18_am.png", + "permalink_public": "https://slack-files.com/T01CWUYP5AR-F0608U3FJ3D-5e001bbbb0", + "is_starred": false, + "has_rich_preview": false, + "file_access": "visible" + }, + { + "id": "F05V8FALBL7", + "created": 1696823970, + "timestamp": 1696823970, + "name": "cluster-logs", + "title": "cluster-logs", + "mimetype": "text/plain", + "filetype": "text", + "pretty_type": "Plain Text", + "user": "U05QL7LN2GH", + "user_team": "T01CWUYP5AR", + "editable": true, + "size": 24284, + "mode": "snippet", + "is_external": false, + "external_type": "", + "is_public": true, + "public_url_shared": false, + "display_as_bot": false, + "username": "", + "url_private": "https://files.slack.com/files-pri/T01CWUYP5AR-F05V8FALBL7/cluster-logs", + "url_private_download": "https://files.slack.com/files-pri/T01CWUYP5AR-F05V8FALBL7/download/cluster-logs", + "permalink": "https://openlineage.slack.com/files/U05QL7LN2GH/F05V8FALBL7/cluster-logs", + "permalink_public": "https://slack-files.com/T01CWUYP5AR-F05V8FALBL7-705f16d652", + "edit_link": "https://openlineage.slack.com/files/U05QL7LN2GH/F05V8FALBL7/cluster-logs/edit", + "preview": "23/10/09 03:53:29 INFO InMemoryFileIndex: Start listing leaf files and directories. Size of Paths: 1; threshold: 32\n23/10/09 03:53:29 INFO InMemoryFileIndex: Start listing leaf files and directories. Size of Paths: 0; threshold: 32\n23/10/09 03:53:29 INFO InMemoryFileIndex: It took 18 ms to list leaf files for 1 paths.\n23/10/09 03:53:29 INFO InMemoryFileIndex: Start listing leaf files and directories. Size of Paths: 1; threshold: 32\n23/10/09 03:53:29 INFO InMemoryFileIndex: Start listing leaf files and directories. Size of Paths: 0; threshold: 32", + "preview_highlight": "
\n
\n
23/10/09 03:53:29 INFO InMemoryFileIndex: Start listing leaf files and directories. Size of Paths: 1; threshold: 32
\n
23/10/09 03:53:29 INFO InMemoryFileIndex: Start listing leaf files and directories. Size of Paths: 0; threshold: 32
\n
23/10/09 03:53:29 INFO InMemoryFileIndex: It took 18 ms to list leaf files for 1 paths.
\n
23/10/09 03:53:29 INFO InMemoryFileIndex: Start listing leaf files and directories. Size of Paths: 1; threshold: 32
\n
23/10/09 03:53:29 INFO InMemoryFileIndex: Start listing leaf files and directories. Size of Paths: 0; threshold: 32
\n
\n
\n", + "lines": 197, + "lines_more": 192, + "preview_is_truncated": true, + "is_starred": false, + "has_rich_preview": false, + "file_access": "visible" + } + ], + "upload": false, + "user": "U05QL7LN2GH", + "ts": "1696823976.297949", + "blocks": [ + { + "type": "rich_text", + "block_id": "2pd86", + "elements": [ + { + "type": "rich_text_section", + "elements": [ + { + "type": "broadcast", + "range": "here" + }, + { + "type": "text", + "text": " I am trying out the openlineage integration of spark on databricks. There is no event getting emitted from Openlineage, I see logs saying OpenLineage Event Skipped. I am attaching the Notebook that i am trying to run and the cluster logs. Kindly can someone help me on this" + } + ] + } + ] + } + ], + "client_msg_id": "b06c052e-bedd-4793-82fb-0dacb289b416", + "thread_ts": "1696823976.297949", + "reply_count": 16, + "reply_users_count": 3, + "latest_reply": "1696843014.053669", + "reply_users": [ + "U05T8BJD4DU", + "U05QL7LN2GH", + "U02MK6YNAQ5" + ], + "is_locked": false, + "subscribed": false, + "replies": [ + { + "client_msg_id": "befe4efc-64c4-4c6b-b7f8-0c9e1f8f5251", + "type": "message", + "text": "from my experience, it will only work on Spark 3.3.x or below, aka Runtime 12.2 or below. Anything above the events will show up once in a blue moon", + "user": "U05T8BJD4DU", + "ts": "1696824130.926979", + "blocks": [ + { + "type": "rich_text", + "block_id": "R6oYJ", + "elements": [ + { + "type": "rich_text_section", + "elements": [ + { + "type": "text", + "text": "from my experience, it will only work on Spark 3.3.x or below, aka Runtime 12.2 or below. Anything above the events will show up once in a blue moon" + } + ] + } + ] + } + ], + "team": "T01CWUYP5AR", + "thread_ts": "1696823976.297949", + "parent_user_id": "U05QL7LN2GH" + }, + { + "client_msg_id": "34df7151-21b7-45f4-b5cc-8d710ca64471", + "type": "message", + "text": "ohh, thanks for the information <@U05T8BJD4DU>, I am trying out with 13.3 Databricks Version and Spark 3.4.1, will try using a below version as you suggested. Any issue tracking this bug <@U05T8BJD4DU>", + "user": "U05QL7LN2GH", + "ts": "1696824278.069699", + "blocks": [ + { + "type": "rich_text", + "block_id": "eW6Pl", + "elements": [ + { + "type": "rich_text_section", + "elements": [ + { + "type": "text", + "text": "ohh, thanks for the information " + }, + { + "type": "user", + "user_id": "U05T8BJD4DU" + }, + { + "type": "text", + "text": ", I am trying out with 13.3 Databricks Version and Spark 3.4.1, will try using a below version as you suggested. Any issue tracking this bug " + }, + { + "type": "user", + "user_id": "U05T8BJD4DU" + } + ] + } + ] + } + ], + "team": "T01CWUYP5AR", + "thread_ts": "1696823976.297949", + "parent_user_id": "U05QL7LN2GH" + }, + { + "client_msg_id": "6f03969d-50fc-46d3-bdb9-e23282b1ac09", + "type": "message", + "text": "", + "user": "U05T8BJD4DU", + "ts": "1696824366.892649", + "blocks": [ + { + "type": "rich_text", + "block_id": "vmCqe", + "elements": [ + { + "type": "rich_text_section", + "elements": [ + { + "type": "link", + "url": "https://github.com/OpenLineage/OpenLineage/issues/2124" + } + ] + } + ] + } + ], + "team": "T01CWUYP5AR", + "attachments": [ + { + "id": 1, + "footer_icon": "https://slack.github.com/static/img/favicon-neutral.png", + "ts": 1695498902, + "color": "36a64f", + "bot_id": "B01VA0FB340", + "app_unfurl_url": "https://github.com/OpenLineage/OpenLineage/issues/2124", + "is_app_unfurl": true, + "app_id": "A01BP7R4KNY", + "fallback": "#2124 Same Delta Table not catching the location on write", + "text": "*What is the target system?*\n\nSpark / Databricks\n\n*What kind of integration is this?*\n\n☐ Produces OpenLineage metadata\n☐ Consumes OpenLineage metadata\n☐ Something else\n\n*How should this integration be implemented?*\n\nI am using OL 1.2.2, Azure Databricks Runtime 11.3 LTS. When creating a table writing into a ADLS location, OL won't be able to catch the location of the output. But when I read the same object it will be able to read the location as INPUT.\n\nPlease note I have also tested Databricks Runtime 13.3 LTS, Spark 3.4.1 - it will give correct ADLS location in INPUT but the input will only show up once in a blue moon. Most of the time the inputs and outputs are blank.\n\n```\n \"inputs\": [],\n \"outputs\": []\n```\n\n```\nCREATE OR REPLACE TABLE transactions_adj\nUSING DELTA LOCATION ''\nAS\n SELECT\n household_id,\n basket_id,\n week_no,\n day,\n transaction_time,\n store_id,\n product_id,\n amount_list,\n campaign_coupon_discount,\n manuf_coupon_discount,\n manuf_coupon_match_discount,\n total_coupon_discount,\n instore_discount,\n amount_paid,\n units\n FROM (\n SELECT \n household_id,\n basket_id,\n week_no,\n day,\n transaction_time,\n store_id,\n product_id,\n COALESCE(sales_amount - discount_amount - coupon_discount - coupon_discount_match,0.0) as amount_list,\n CASE \n WHEN COALESCE(coupon_discount_match,0.0) = 0.0 THEN -1 * COALESCE(coupon_discount,0.0) \n ELSE 0.0 \n END as campaign_coupon_discount,\n CASE \n WHEN COALESCE(coupon_discount_match,0.0) != 0.0 THEN -1 * COALESCE(coupon_discount,0.0) \n ELSE 0.0 \n END as manuf_coupon_discount,\n -1 * COALESCE(coupon_discount_match,0.0) as manuf_coupon_match_discount,\n -1 * COALESCE(coupon_discount - coupon_discount_match,0.0) as total_coupon_discount,\n COALESCE(-1 * discount_amount,0.0) as instore_discount,\n COALESCE(sales_amount,0.0) as `amount_paid,`\n quantity as units\n FROM transactions\n );\n```\n\nHere's the COMPLETE event:\n\n```\n\n \"outputs\":[\n {\n \"namespace\":\"dbfs\",\n \"name\":\"/user/hive/warehouse/journey.db/transactions_adj\",\n \"facets\":{\n \"dataSource\":{\n \"_producer\":\"\",\n \"_schemaURL\":\"\",\n \"name\":\"dbfs\",\n \"uri\":\"dbfs\"\n },\n\n```\n\nBelow logical plan shows the path:\n\n```\n== Analyzed Logical Plan ==\nnum_affected_rows: bigint, num_inserted_rows: bigint\nReplaceTableAsSelect TableSpec(Map(),Some(DELTA),Map(),Some(),None,None,false,Set()), true\n:- ResolvedIdentifier com.databricks.sql.managedcatalog.UnityCatalogV2Proxy@6251a8df, default.transactions_adj\n+- Project [household_id#184, basket_id#185L, week_no#193, day#186, transaction_time#192, store_id#190, product_id#187, amount_list#147, campaign_coupon_discount#148, manuf_coupon_discount#149, manuf_coupon_match_discount#150, total_coupon_discount#151, instore_discount#152, amount_paid#153, units#154]\n +- SubqueryAlias __auto_generated_subquery_name\n +- Project [household_id#184, basket_id#185L, week_no#193, day#186, transaction_time#192, store_id#190, product_id#187, coalesce(cast((((sales_amount#189 - discount_amount#191) - coupon_discount#194) - coupon_discount_match#195) as double), cast(0.0 as double)) AS amount_list#147, CASE WHEN (coalesce(cast(coupon_discount_match#195 as double), cast(0.0 as double)) = cast(0.0 as double)) THEN (cast(-1 as double) * coalesce(cast(coupon_discount#194 as double), cast(0.0 as double))) ELSE cast(0.0 as double) END AS campaign_coupon_discount#148, CASE WHEN NOT (coalesce(cast(coupon_discount_match#195 as double), cast(0.0 as double)) = cast(0.0 as double)) THEN (cast(-1 as double) * coalesce(cast(coupon_discount#194 as double), cast(0.0 as double))) ELSE cast(0.0 as double) END AS manuf_coupon_discount#149, (cast(-1 as double) * coalesce(cast(coupon_discount_match#195 as double), cast(0.0 as double))) AS manuf_coupon_match_discount#150, (cast(-1 as double) * coalesce(cast((coupon_discount#194 - coupon_discount_match#195) as double), cast(0.0 as double))) AS total_coupon_discount#151, coalesce(cast((cast(-1 as float) * discount_amount#191) as double), cast(0.0 as double)) AS instore_discount#152, coalesce(cast(sales_amount#189 as double), cast(0.0 as double)) AS amount_paid#153, quantity#188 AS units#154]\n +- SubqueryAlias spark_catalog.default.transactions\n +- Relation spark_catalog.default.transactions[household_id#184,basket_id#185L,day#186,product_id#187,quantity#188,sales_amount#189,store_id#190,discount_amount#191,transaction_time#192,week_no#193,coupon_discount#194,coupon_discount_match#195] parquet\n```\n\n*Where should this integration be implemented?*\n\n☐ In the target system\n☐ In the OpenLineage repo\n☐ Somewhere else\n\n*Do you plan to make this contribution yourself?*\n\n☐ I am interested in doing this work", + "title": "#2124 Same Delta Table not catching the location on write", + "title_link": "https://github.com/OpenLineage/OpenLineage/issues/2124", + "footer": "", + "fields": [ + { + "value": "integration/spark, integration/databricks", + "title": "Labels", + "short": true + }, + { + "value": "2", + "title": "Comments", + "short": true + } + ], + "mrkdwn_in": [ + "text" + ] + } + ], + "thread_ts": "1696823976.297949", + "parent_user_id": "U05QL7LN2GH" + }, + { + "client_msg_id": "ed33723b-e22c-40a2-9799-de7dd1030a5e", + "type": "message", + "text": "tried with databricks 12.2 --> spark 3.3.2, still the same behaviour no event getting emitted", + "user": "U05QL7LN2GH", + "ts": "1696824714.873719", + "blocks": [ + { + "type": "rich_text", + "block_id": "m4glh", + "elements": [ + { + "type": "rich_text_section", + "elements": [ + { + "type": "text", + "text": "tried with databricks 12.2 --> spark 3.3.2, still the same behaviour no event getting emitted" + } + ] + } + ] + } + ], + "team": "T01CWUYP5AR", + "edited": { + "user": "U05QL7LN2GH", + "ts": "1696824726.000000" + }, + "thread_ts": "1696823976.297949", + "parent_user_id": "U05QL7LN2GH" + }, + { + "client_msg_id": "188281e1-7fa2-4ae6-a139-e873476ef677", + "type": "message", + "text": "you can do 11.3, its the most stable one I know", + "user": "U05T8BJD4DU", + "ts": "1696824755.134769", + "blocks": [ + { + "type": "rich_text", + "block_id": "Obry3", + "elements": [ + { + "type": "rich_text_section", + "elements": [ + { + "type": "text", + "text": "you can do 11.3, its the most stable one I know" + } + ] + } + ] + } + ], + "team": "T01CWUYP5AR", + "thread_ts": "1696823976.297949", + "parent_user_id": "U05QL7LN2GH" + }, + { + "client_msg_id": "bec85693-dc8e-4bb3-8ab7-a5e41817b485", + "type": "message", + "text": "sure, let me try that out", + "user": "U05QL7LN2GH", + "ts": "1696824766.361999", + "blocks": [ + { + "type": "rich_text", + "block_id": "6qxi+", + "elements": [ + { + "type": "rich_text_section", + "elements": [ + { + "type": "text", + "text": "sure, let me try that out" + } + ] + } + ] + } + ], + "team": "T01CWUYP5AR", + "thread_ts": "1696823976.297949", + "parent_user_id": "U05QL7LN2GH" + }, + { + "client_msg_id": "95ca842b-8bc8-4158-89ce-7332950b6202", + "type": "message", + "text": "still the same problem…the jar that i am using is the latest _openlineage-spark-1.3.1.jar, do you think that can be the problem_", + "user": "U05QL7LN2GH", + "ts": "1696825911.714029", + "blocks": [ + { + "type": "rich_text", + "block_id": "5kxtF", + "elements": [ + { + "type": "rich_text_section", + "elements": [ + { + "type": "text", + "text": "still the same problem…the jar that i am using is the latest " + }, + { + "type": "text", + "text": "openlineage-spark-1.3.1.jar, do you think that can be the problem", + "style": { + "italic": true + } + } + ] + } + ] + } + ], + "team": "T01CWUYP5AR", + "edited": { + "user": "U05QL7LN2GH", + "ts": "1696825923.000000" + }, + "thread_ts": "1696823976.297949", + "parent_user_id": "U05QL7LN2GH" + }, + { + "client_msg_id": "7ace5bd1-ec85-4f9c-835a-580615adf030", + "type": "message", + "text": "tried with _openlineage-spark-1.2.2.jar, still the same issue, seems like they are skipping some events_", + "user": "U05QL7LN2GH", + "ts": "1696826639.326199", + "blocks": [ + { + "type": "rich_text", + "block_id": "4H36W", + "elements": [ + { + "type": "rich_text_section", + "elements": [ + { + "type": "text", + "text": "tried with " + }, + { + "type": "text", + "text": "openlineage-spark-1.2.2.jar, still the same issue, seems like they are skipping some events", + "style": { + "italic": true + } + } + ] + } + ] + } + ], + "team": "T01CWUYP5AR", + "thread_ts": "1696823976.297949", + "parent_user_id": "U05QL7LN2GH" + }, + { + "client_msg_id": "d33daeb6-44d9-4759-9796-2196663d0791", + "type": "message", + "text": "Probably not all events will be captured, I have only tested create tables and jobs", + "user": "U05T8BJD4DU", + "ts": "1696830440.910309", + "blocks": [ + { + "type": "rich_text", + "block_id": "WJ2BZ", + "elements": [ + { + "type": "rich_text_section", + "elements": [ + { + "type": "text", + "text": "Probably not all events will be captured, I have only tested create tables and jobs" + } + ] + } + ] + } + ], + "team": "T01CWUYP5AR", + "thread_ts": "1696823976.297949", + "parent_user_id": "U05QL7LN2GH" + }, + { + "client_msg_id": "f31b094c-55cd-46d7-bf16-1a14b11df963", + "type": "message", + "text": "Hi <@U05QL7LN2GH>, how did you configure openlineage and what is your job doing?\n\nWe do have a bunch of integration tests on Databricks platform and they're passing on databricks runtime `13.0.x-scala2.12`.\n\nCould you also try running code same as our test does ()? If you run it and see OL events, this will make us sure your config is OK and we can continue further debug.\n\nLooking at your spark script: could you save your dataset and see if you still don't see any events?", + "user": "U02MK6YNAQ5", + "ts": "1696840272.582079", + "blocks": [ + { + "type": "rich_text", + "block_id": "lq3NX", + "elements": [ + { + "type": "rich_text_section", + "elements": [ + { + "type": "text", + "text": "Hi " + }, + { + "type": "user", + "user_id": "U05QL7LN2GH" + }, + { + "type": "text", + "text": ", how did you configure openlineage and what is your job doing?\n\nWe do have a bunch of integration tests on Databricks platform " + }, + { + "type": "link", + "url": "https://github.com/OpenLineage/OpenLineage/blob/1.3.1/integration/spark/app/src/test/java/io/openlineage/spark/agent/DatabricksIntegrationTest.java", + "text": "available here" + }, + { + "type": "text", + "text": " and they're passing on databricks runtime " + }, + { + "type": "text", + "text": "13.0.x-scala2.12", + "style": { + "code": true + } + }, + { + "type": "text", + "text": ".\n\nCould you also try running code same as our test does (" + }, + { + "type": "link", + "url": "https://github.com/OpenLineage/OpenLineage/blob/1.3.1/integration/spark/app/src/test/resources/databricks_notebooks/ctas.py", + "text": "this one" + }, + { + "type": "text", + "text": ")? If you run it and see OL events, this will make us sure your config is OK and we can continue further debug.\n\nLooking at your spark script: could you save your dataset and see if you still don't see any events?" + } + ] + } + ] + } + ], + "team": "T01CWUYP5AR", + "attachments": [ + { + "id": 1, + "footer_icon": "https://slack.github.com/static/img/favicon-neutral.png", + "color": "24292f", + "bot_id": "B01VA0FB340", + "app_unfurl_url": "https://github.com/OpenLineage/OpenLineage/blob/1.3.1/integration/spark/app/src/test/java/io/openlineage/spark/agent/DatabricksIntegrationTest.java", + "is_app_unfurl": true, + "app_id": "A01BP7R4KNY", + "fallback": "", + "title": "", + "footer": "", + "mrkdwn_in": [ + "text" + ] + }, + { + "id": 2, + "footer_icon": "https://slack.github.com/static/img/favicon-neutral.png", + "color": "24292f", + "bot_id": "B01VA0FB340", + "app_unfurl_url": "https://github.com/OpenLineage/OpenLineage/blob/1.3.1/integration/spark/app/src/test/resources/databricks_notebooks/ctas.py", + "is_app_unfurl": true, + "app_id": "A01BP7R4KNY", + "fallback": "", + "title": "", + "footer": "", + "mrkdwn_in": [ + "text" + ] + } + ], + "thread_ts": "1696823976.297949", + "parent_user_id": "U05QL7LN2GH" + }, + { + "client_msg_id": "25205ec0-ce7f-48fc-976d-24e50726d962", + "type": "message", + "text": "```babynames = spark.read.format(\"csv\").option(\"header\", \"true\").option(\"inferSchema\", \"true\").load(\"dbfs:/FileStore/babynames.csv\")\nbabynames.createOrReplaceTempView(\"babynames_table\")\nyears = spark.sql(\"select distinct(Year) from babynames_table\").rdd.map(lambda row : row[0]).collect()\nyears.sort()\ndbutils.widgets.dropdown(\"year\", \"2014\", [str(x) for x in years])\ndisplay(babynames.filter(babynames.Year == dbutils.widgets.get(\"year\")))```", + "user": "U05QL7LN2GH", + "ts": "1696842401.831669", + "blocks": [ + { + "type": "rich_text", + "block_id": "gtFWb", + "elements": [ + { + "type": "rich_text_preformatted", + "elements": [ + { + "type": "text", + "text": "babynames = spark.read.format(\"csv\").option(\"header\", \"true\").option(\"inferSchema\", \"true\").load(\"dbfs:/FileStore/babynames.csv\")\nbabynames.createOrReplaceTempView(\"babynames_table\")\nyears = spark.sql(\"select distinct(Year) from babynames_table\").rdd.map(lambda row : row[0]).collect()\nyears.sort()\ndbutils.widgets.dropdown(\"year\", \"2014\", [str(x) for x in years])\ndisplay(babynames.filter(babynames.Year == dbutils.widgets.get(\"year\")))" + } + ], + "border": 0 + } + ] + } + ], + "team": "T01CWUYP5AR", + "thread_ts": "1696823976.297949", + "parent_user_id": "U05QL7LN2GH" + }, + { + "client_msg_id": "47f38de3-3b70-4221-9aee-d900c38e2b46", + "type": "message", + "text": "this is the script that i am running <@U02MK6YNAQ5>…kindly let me know if i’m doing any mistake. I have added the init script at the cluster level and from the logs i could see that openlineage is configured as i see a log statement", + "user": "U05QL7LN2GH", + "ts": "1696842489.837109", + "blocks": [ + { + "type": "rich_text", + "block_id": "TAZ4O", + "elements": [ + { + "type": "rich_text_section", + "elements": [ + { + "type": "text", + "text": "this is the script that i am running " + }, + { + "type": "user", + "user_id": "U02MK6YNAQ5" + }, + { + "type": "text", + "text": "…kindly let me know if i’m doing any mistake. I have added the init script at the cluster level and from the logs i could see that openlineage is configured as i see a log statement" + } + ] + } + ] + } + ], + "team": "T01CWUYP5AR", + "thread_ts": "1696823976.297949", + "parent_user_id": "U05QL7LN2GH" + }, + { + "client_msg_id": "0c59edb2-d1a5-4d73-bc86-c3f5d8195a99", + "type": "message", + "text": "there's nothing wrong in that script. It's just we decided to limit amount of OL events for jobs that don't write their data anywhere and just do `collect` operation", + "user": "U02MK6YNAQ5", + "ts": "1696842630.801999", + "blocks": [ + { + "type": "rich_text", + "block_id": "7MwlU", + "elements": [ + { + "type": "rich_text_section", + "elements": [ + { + "type": "text", + "text": "there's nothing wrong in that script. It's just we decided to limit amount of OL events for jobs that don't write their data anywhere and just do " + }, + { + "type": "text", + "text": "collect", + "style": { + "code": true + } + }, + { + "type": "text", + "text": " operation" + } + ] + } + ] + } + ], + "team": "T01CWUYP5AR", + "thread_ts": "1696823976.297949", + "parent_user_id": "U05QL7LN2GH" + }, + { + "client_msg_id": "d6540e9a-2c67-494e-8362-fd911feda346", + "type": "message", + "text": "this is also a potential reason why can't you see any events", + "user": "U02MK6YNAQ5", + "ts": "1696842662.946709", + "blocks": [ + { + "type": "rich_text", + "block_id": "RjYDU", + "elements": [ + { + "type": "rich_text_section", + "elements": [ + { + "type": "text", + "text": "this is also a potential reason why can't you see any events" + } + ] + } + ] + } + ], + "team": "T01CWUYP5AR", + "thread_ts": "1696823976.297949", + "parent_user_id": "U05QL7LN2GH" + }, + { + "client_msg_id": "f7360d94-2779-478b-8a55-bd6c36c10ad6", + "type": "message", + "text": "ohh…okk, will try out the test script that you have mentioned above. Kindly correct me if my understanding is correct, so if there are a few transformatiosna nd finally writing somewhere that is where the OL events are expected to be emitted?", + "user": "U05QL7LN2GH", + "ts": "1696842873.465329", + "blocks": [ + { + "type": "rich_text", + "block_id": "575as", + "elements": [ + { + "type": "rich_text_section", + "elements": [ + { + "type": "text", + "text": "ohh…okk, will try out the test script that you have mentioned above. Kindly correct me if my understanding is correct, so if there are a few transformatiosna nd finally writing somewhere that is where the OL events are expected to be emitted?" + } + ] + } + ] + } + ], + "team": "T01CWUYP5AR", + "thread_ts": "1696823976.297949", + "parent_user_id": "U05QL7LN2GH" + }, + { + "client_msg_id": "d7d6a2b8-8149-43bc-a904-b2fd82116e4b", + "type": "message", + "text": "yes. main purpose of the lineage is to track dependencies between the datasets, when a job reads from dataset A and writes to dataset B. In case of databricks notebook, that do `show` or `collect` and print some query result on the screen, there may be no reason to track it in the sense of lineage.", + "user": "U02MK6YNAQ5", + "ts": "1696843014.053669", + "blocks": [ + { + "type": "rich_text", + "block_id": "NZ3aU", + "elements": [ + { + "type": "rich_text_section", + "elements": [ + { + "type": "text", + "text": "yes. main purpose of the lineage is to track dependencies between the datasets, when a job reads from dataset A and writes to dataset B. In case of databricks notebook, that do " + }, + { + "type": "text", + "text": "show", + "style": { + "code": true + } + }, + { + "type": "text", + "text": " or " + }, + { + "type": "text", + "text": "collect", + "style": { + "code": true + } + }, + { + "type": "text", + "text": " and print some query result on the screen, there may be no reason to track it in the sense of lineage." + } + ] + } + ] + } + ], + "team": "T01CWUYP5AR", + "thread_ts": "1696823976.297949", + "parent_user_id": "U05QL7LN2GH" + } + ] + }, + { + "client_msg_id": "bc8de3a0-c11b-4ac1-9d0c-b2d67771f442", + "type": "message", + "text": "<@U02LXF3HUN7> can we cut a new release to include this change?\n• ", + "user": "U01HVNU6A4C", + "ts": "1696591141.778179", + "blocks": [ + { + "type": "rich_text", + "block_id": "DOR9k", + "elements": [ + { + "type": "rich_text_section", + "elements": [ + { + "type": "user", + "user_id": "U02LXF3HUN7" + }, + { + "type": "text", + "text": " can we cut a new release to include this change?\n" + } + ] + }, + { + "type": "rich_text_list", + "elements": [ + { + "type": "rich_text_section", + "elements": [ + { + "type": "link", + "url": "https://github.com/OpenLineage/OpenLineage/pull/2151" + } + ] + } + ], + "style": "bullet", + "indent": 0, + "border": 0 + } + ] + } + ], + "team": "T01CWUYP5AR", + "attachments": [ + { + "id": 1, + "footer_icon": "https://slack.github.com/static/img/favicon-neutral.png", + "ts": 1696337382, + "color": "6f42c1", + "bot_id": "B01VA0FB340", + "app_unfurl_url": "https://github.com/OpenLineage/OpenLineage/pull/2151", + "is_app_unfurl": true, + "app_id": "A01BP7R4KNY", + "fallback": "#2151 Allow setting client's endpoint via environment variable", + "text": "*Problem*\n\nCurrently, it's not possible to set the OpenLineage endpoint (hard-coded to `/api/v1/lineage`) using an environment variable when running the Airflow integration.\n\n*Solution*\n\nGiven that it's not possible to create the client manually in Airflow, especially now that OpenLineage has become an official Airflow provider, this change seems like the only feasible solution.\n\n☐ Your change modifies the OpenLineage model\n☐ Your change modifies one or more OpenLineage \n\n*One-line summary:*\n\nAllow setting client's endpoint via environment variable.\n\n*Checklist*\n\n☑︎ You've your work\n☑︎ Your pull request title follows our \n☑︎ Your changes are accompanied by tests (_if relevant_)\n☑︎ Your change contains a and is self-contained\n☑︎ You've updated any relevant documentation (_if relevant_)\n☑︎ Your comment includes a one-liner for the changelog about the specific purpose of the change (_if necessary_)\n☐ You've versioned the core OpenLineage model or facets according to (_if relevant_)\n☐ You've added a to source files (_if relevant_)\n\n* * *\n\nSPDX-License-Identifier: Apache-2.0 \nCopyright 2018-2023 contributors to the OpenLineage project", + "title": "#2151 Allow setting client's endpoint via environment variable", + "title_link": "https://github.com/OpenLineage/OpenLineage/pull/2151", + "footer": "", + "fields": [ + { + "value": "documentation, client/python", + "title": "Labels", + "short": true + }, + { + "value": "6", + "title": "Comments", + "short": true + } + ], + "mrkdwn_in": [ + "text" + ] + } + ], + "thread_ts": "1696591141.778179", + "reply_count": 1, + "reply_users_count": 1, + "latest_reply": "1696634162.280029", + "reply_users": [ + "U02LXF3HUN7" + ], + "is_locked": false, + "subscribed": true, + "last_read": "1696634162.280029", + "reactions": [ + { + "name": "heavy_plus_sign", + "users": [ + "U01HNKK4XAM", + "U02S6F54MAB", + "U01DCLP0GU9", + "U02LXF3HUN7", + "U01RA9B5GG2" + ], + "count": 5 + } + ], + "replies": [ + { + "client_msg_id": "24ac081b-fae5-447d-8767-8b560faa18db", + "type": "message", + "text": "Thanks for requesting a release, <@U01HVNU6A4C>. It has been approved and will be initiated within 2 business days of next Monday.", + "user": "U02LXF3HUN7", + "ts": "1696634162.280029", + "blocks": [ + { + "type": "rich_text", + "block_id": "IA9Xp", + "elements": [ + { + "type": "rich_text_section", + "elements": [ + { + "type": "text", + "text": "Thanks for requesting a release, " + }, + { + "type": "user", + "user_id": "U01HVNU6A4C" + }, + { + "type": "text", + "text": ". It has been approved and will be initiated within 2 business days of next Monday." + } + ] + } + ] + } + ], + "team": "T01CWUYP5AR", + "thread_ts": "1696591141.778179", + "parent_user_id": "U01HVNU6A4C", + "reactions": [ + { + "name": "pray", + "users": [ + "U01HVNU6A4C" + ], + "count": 1 + } + ] + } + ] + }, + { + "client_msg_id": "D5AAA7E8-6425-4534-B01B-2BB732AD3E31", + "type": "message", + "text": "The Marquez meetup in San Francisco is happening right now!\n", + "user": "U01DCLP0GU9", + "ts": "1696552840.350759", + "blocks": [ + { + "type": "rich_text", + "block_id": "MOzWE", + "elements": [ + { + "type": "rich_text_section", + "elements": [ + { + "type": "text", + "text": "The Marquez meetup in San Francisco is happening right now!\n" + }, + { + "type": "link", + "url": "https://www.meetup.com/meetup-group-bnfqymxe/events/295444209/?utm_medium=referral&utm_campaign=share-btn_savedevents_share_modal&utm_source=link", + "text": "https://www.meetup.com/meetup-group-bnfqymxe/events/295444209/?utm_medium=referral&utm_campaign=share-btn_savedevents_share_modal&utm_source=link" + } + ] + } + ] + } + ], + "team": "T01CWUYP5AR", + "edited": { + "user": "U01DCLP0GU9", + "ts": "1696553106.000000" + }, + "attachments": [ + { + "image_url": "https://secure.meetupstatic.com/photos/event/a/1/8/c/600_515141356.jpeg", + "image_width": 600, + "image_height": 338, + "image_bytes": 12395, + "from_url": "https://www.meetup.com/meetup-group-bnfqymxe/events/295444209/?utm_medium=referral&utm_campaign=share-btn_savedevents_share_modal&utm_source=link", + "service_icon": "https://secure.meetupstatic.com/next/images/general/m_swarm_120x120.png", + "id": 1, + "original_url": "https://www.meetup.com/meetup-group-bnfqymxe/events/295444209/?utm_medium=referral&utm_campaign=share-btn_savedevents_share_modal&utm_source=link", + "fallback": "Meetup: Marquez Meetup @ Astronomer, Thu, Oct 5, 2023, 5:30 PM | Meetup", + "text": "Join us on Thursday, October 5th, from 5:30-8:30 pm to learn about the Marquez project. Meet other members of the community, get tips on making the most of the latest impro", + "title": "Marquez Meetup @ Astronomer, Thu, Oct 5, 2023, 5:30 PM | Meetup", + "title_link": "https://www.meetup.com/meetup-group-bnfqymxe/events/295444209/?utm_medium=referral&utm_campaign=share-btn_savedevents_share_modal&utm_source=link", + "service_name": "Meetup" + } + ], + "reactions": [ + { + "name": "tada", + "users": [ + "U02MK6YNAQ5", + "U05TU0U224A" + ], + "count": 2 + } + ] + }, + { + "type": "message", + "subtype": "thread_broadcast", + "text": "I have created a ticket to make this easier to find. Once I get more feedback I’ll turn it into a md file in the repo: \n", + "user": "U01DCLP0GU9", + "ts": "1696541652.452819", + "thread_ts": "1694737381.437569", + "root": { + "client_msg_id": "32e40b58-9b35-45a3-99aa-e37404cd6329", + "type": "message", + "text": "Per discussion in the OpenLineage sync today here is a very early strawman proposal for an OpenLineage registry that producers and consumers could be registered in.\nFeedback or alternate proposals welcome\n\nOnce this is sufficiently fleshed out, I’ll create an actual proposal on github", + "user": "U01DCLP0GU9", + "ts": "1694737381.437569", + "blocks": [ + { + "type": "rich_text", + "block_id": "KKjtL", + "elements": [ + { + "type": "rich_text_section", + "elements": [ + { + "type": "text", + "text": "Per discussion in the OpenLineage sync today here is a very early strawman proposal for an OpenLineage registry that producers and consumers could be registered in.\nFeedback or alternate proposals welcome\n" + }, + { + "type": "link", + "url": "https://docs.google.com/document/d/1zIxKST59q3I6ws896M4GkUn7IsueLw8ejct5E-TR0vY/edit" + }, + { + "type": "text", + "text": "\nOnce this is sufficiently fleshed out, I’ll create an actual proposal on github" + } + ] + } + ] + } + ], + "team": "T01CWUYP5AR", + "thread_ts": "1694737381.437569", + "reply_count": 2, + "reply_users_count": 1, + "latest_reply": "1696541652.452819", + "reply_users": [ + "U01DCLP0GU9" + ], + "is_locked": false, + "subscribed": false + }, + "blocks": [ + { + "type": "rich_text", + "block_id": "EbQGP", + "elements": [ + { + "type": "rich_text_section", + "elements": [ + { + "type": "text", + "text": "I have created a ticket to make this easier to find. Once I get more feedback I’ll turn it into a md file in the repo: " + }, + { + "type": "link", + "url": "https://docs.google.com/document/d/1zIxKST59q3I6ws896M4GkUn7IsueLw8ejct5E-TR0vY/edit#heading=h.enpbmvu7n8gu" + }, + { + "type": "text", + "text": "\n" + }, + { + "type": "link", + "url": "https://github.com/OpenLineage/OpenLineage/issues/2161" + } + ] + } + ] + } + ], + "attachments": [ + { + "id": 1, + "footer_icon": "https://slack.github.com/static/img/favicon-neutral.png", + "ts": 1696541261, + "color": "36a64f", + "bot_id": "B01VA0FB340", + "app_unfurl_url": "https://github.com/OpenLineage/OpenLineage/issues/2161", + "is_app_unfurl": true, + "app_id": "A01BP7R4KNY", + "fallback": "#2161 [PROPOSAL] Add a Registry of Producers and Consumers in OpenLineage", + "text": "*Purpose*\n\nThis is the early stage of an idea to get community feedback on what an OpenLineage registry for producers, custom facets and consumers could be. Once this document is stable enough, I’ll create an official proposal on the OpenLineage repo.\n\n*Goal*\n\nAllow third parties to register their implementations or custom extensions to make them easy to discover. \nShorten “Producer” and “schema url” values\n\n*Proposed implementation*\n\nCurrent draft for discussion:\n\n", + "title": "#2161 [PROPOSAL] Add a Registry of Producers and Consumers in OpenLineage", + "title_link": "https://github.com/OpenLineage/OpenLineage/issues/2161", + "footer": "", + "fields": [ + { + "value": "proposal", + "title": "Labels", + "short": true + } + ], + "mrkdwn_in": [ + "text" + ] + } + ], + "client_msg_id": "b2b64b49-4c25-427f-b1ae-f1fa92aad028", + "edited": { + "user": "U01DCLP0GU9", + "ts": "1696541673.000000" + } + }, + { + "client_msg_id": "3406f12f-2796-42b3-8c6a-37131a83311e", + "type": "message", + "text": "**\nThis month’s TSC meeting is next Thursday the 12th at 10am PT. On the tentative agenda:\n• announcements\n• recent releases\n• Airflow Summit recap\n• tutorial: migrating to the Airflow Provider\n• discussion topic: observability for OpenLineage/Marquez\n• open discussion\n• more (TBA)\nMore info and the meeting link can be found on the . All are welcome! Do you have a discussion topic, use case or integration you’d like to demo? DM me to be added to the agenda.", + "user": "U02LXF3HUN7", + "ts": "1696531454.431629", + "blocks": [ + { + "type": "rich_text", + "block_id": "TBGz6", + "elements": [ + { + "type": "rich_text_section", + "elements": [ + { + "type": "broadcast", + "range": "channel", + "style": { + "bold": true + } + }, + { + "type": "text", + "text": "\nThis month’s TSC meeting is next Thursday the 12th at 10am PT. On the tentative agenda:\n" + } + ] + }, + { + "type": "rich_text_list", + "elements": [ + { + "type": "rich_text_section", + "elements": [ + { + "type": "text", + "text": "announcements" + } + ] + }, + { + "type": "rich_text_section", + "elements": [ + { + "type": "text", + "text": "recent releases" + } + ] + }, + { + "type": "rich_text_section", + "elements": [ + { + "type": "text", + "text": "Airflow Summit recap" + } + ] + }, + { + "type": "rich_text_section", + "elements": [ + { + "type": "text", + "text": "tutorial: migrating to the Airflow Provider" + } + ] + }, + { + "type": "rich_text_section", + "elements": [ + { + "type": "text", + "text": "discussion topic: observability for OpenLineage/Marquez" + } + ] + }, + { + "type": "rich_text_section", + "elements": [ + { + "type": "text", + "text": "open discussion" + } + ] + }, + { + "type": "rich_text_section", + "elements": [ + { + "type": "text", + "text": "more (TBA)" + } + ] + } + ], + "style": "bullet", + "indent": 0, + "border": 0 + }, + { + "type": "rich_text_section", + "elements": [ + { + "type": "text", + "text": "More info and the meeting link can be found on the " + }, + { + "type": "link", + "url": "https://openlineage.io/meetings/", + "text": "website" + }, + { + "type": "text", + "text": ". All are welcome! Do you have a discussion topic, use case or integration you’d like to demo? DM me to be added to the agenda." + } + ] + } + ] + } + ], + "team": "T01CWUYP5AR", + "edited": { + "user": "U02LXF3HUN7", + "ts": "1696888845.000000" + }, + "attachments": [ + { + "from_url": "https://openlineage.io/meetings/", + "service_icon": "https://openlineage.io/img/favicon.ico", + "id": 1, + "original_url": "https://openlineage.io/meetings/", + "fallback": "TSC Meetings | OpenLineage", + "text": "The OpenLineage Technical Steering Committee meets monthly, and is open to all.", + "title": "TSC Meetings | OpenLineage", + "title_link": "https://openlineage.io/meetings/", + "service_name": "openlineage.io" + } + ], + "reactions": [ + { + "name": "eyes", + "users": [ + "U0323HG8C8H", + "U0544QC1DS9", + "U01SW738WCF" + ], + "count": 3 + } + ] + }, + { + "type": "message", + "subtype": "thread_broadcast", + "text": "I have cleaned up the registry proposal.\n\nIn particular:\n• I clarified that option 2 is preferred at this point.\n• I moved discussion notes to the bottom. they will go away at some point\n• Once it is stable, I’ll create a with the preferred option.\n• we need a good proposal for the core facets prefix. My suggestion is to move core facets to `core` in the registry. The drawback is prefix would be inconsistent.\n", + "user": "U01DCLP0GU9", + "ts": "1696379615.265919", + "thread_ts": "1694737381.437569", + "root": { + "client_msg_id": "32e40b58-9b35-45a3-99aa-e37404cd6329", + "type": "message", + "text": "Per discussion in the OpenLineage sync today here is a very early strawman proposal for an OpenLineage registry that producers and consumers could be registered in.\nFeedback or alternate proposals welcome\n\nOnce this is sufficiently fleshed out, I’ll create an actual proposal on github", + "user": "U01DCLP0GU9", + "ts": "1694737381.437569", + "blocks": [ + { + "type": "rich_text", + "block_id": "KKjtL", + "elements": [ + { + "type": "rich_text_section", + "elements": [ + { + "type": "text", + "text": "Per discussion in the OpenLineage sync today here is a very early strawman proposal for an OpenLineage registry that producers and consumers could be registered in.\nFeedback or alternate proposals welcome\n" + }, + { + "type": "link", + "url": "https://docs.google.com/document/d/1zIxKST59q3I6ws896M4GkUn7IsueLw8ejct5E-TR0vY/edit" + }, + { + "type": "text", + "text": "\nOnce this is sufficiently fleshed out, I’ll create an actual proposal on github" + } + ] + } + ] + } + ], + "team": "T01CWUYP5AR", + "thread_ts": "1694737381.437569", + "reply_count": 2, + "reply_users_count": 1, + "latest_reply": "1696541652.452819", + "reply_users": [ + "U01DCLP0GU9" + ], + "is_locked": false, + "subscribed": false + }, + "blocks": [ + { + "type": "rich_text", + "block_id": "UD6d9", + "elements": [ + { + "type": "rich_text_section", + "elements": [ + { + "type": "text", + "text": "I have cleaned up the registry proposal.\n" + }, + { + "type": "link", + "url": "https://docs.google.com/document/d/1zIxKST59q3I6ws896M4GkUn7IsueLw8ejct5E-TR0vY/edit" + }, + { + "type": "text", + "text": "\nIn particular:\n" + } + ] + }, + { + "type": "rich_text_list", + "elements": [ + { + "type": "rich_text_section", + "elements": [ + { + "type": "text", + "text": "I clarified that option 2 is preferred at this point." + } + ] + }, + { + "type": "rich_text_section", + "elements": [ + { + "type": "text", + "text": "I moved discussion notes to the bottom. they will go away at some point" + } + ] + }, + { + "type": "rich_text_section", + "elements": [ + { + "type": "text", + "text": "Once it is stable, I’ll create a " + }, + { + "type": "link", + "url": "https://github.com/OpenLineage/OpenLineage/tree/main/proposals", + "text": "proposal" + }, + { + "type": "text", + "text": " with the preferred option." + } + ] + }, + { + "type": "rich_text_section", + "elements": [ + { + "type": "text", + "text": "we need a good proposal for the core facets prefix. My suggestion is to move core facets to " + }, + { + "type": "text", + "text": "core", + "style": { + "code": true + } + }, + { + "type": "text", + "text": " in the registry. The drawback is prefix would be inconsistent." + } + ] + } + ], + "style": "bullet", + "indent": 0, + "border": 0 + }, + { + "type": "rich_text_section", + "elements": [] + } + ] + } + ], + "client_msg_id": "4f7a5bb9-269f-4e6d-98b4-669b2760c1bf" + }, + { + "client_msg_id": "dba3fb6e-73e4-4aad-857f-f45506333a4d", + "type": "message", + "text": "Hey everyone - does anyone have a good mechanism for alerting on issues with open lineage? For example, maybe alerting when an event times out - perhaps to prometheus or some other kind of generic endpoint? Not sure the best approach here (if the meta inf extension would be able to achieve it)", + "user": "U03D8K119LJ", + "ts": "1696350897.139129", + "blocks": [ + { + "type": "rich_text", + "block_id": "LXBVs", + "elements": [ + { + "type": "rich_text_section", + "elements": [ + { + "type": "text", + "text": "Hey everyone - does anyone have a good mechanism for alerting on issues with open lineage? For example, maybe alerting when an event times out - perhaps to prometheus or some other kind of generic endpoint? Not sure the best approach here (if the meta inf extension would be able to achieve it)" + } + ] + } + ] + } + ], + "team": "T01CWUYP5AR", + "thread_ts": "1696350897.139129", + "reply_count": 1, + "reply_users_count": 1, + "latest_reply": "1696402862.520309", + "reply_users": [ + "U02MK6YNAQ5" + ], + "is_locked": false, + "subscribed": false, + "replies": [ + { + "client_msg_id": "f2d5f6c3-57ee-4ef2-88b0-6cf59d7f2505", + "type": "message", + "text": "That's a great usecase for OpenLineage. Unfortunately, we don't have any doc or recomendation on that.\n\nI would try using FluentD proxy we have () to copy event stream (alerting is just one of usecases for lineage events) and write fluentd plugin to send it asynchronously further to alerting service like PagerDuty.\n\nIt looks cool to me but I never had enough time to test this approach.", + "user": "U02MK6YNAQ5", + "ts": "1696402862.520309", + "blocks": [ + { + "type": "rich_text", + "block_id": "6FxCH", + "elements": [ + { + "type": "rich_text_section", + "elements": [ + { + "type": "text", + "text": "That's a great usecase for OpenLineage. Unfortunately, we don't have any doc or recomendation on that.\n\nI would try using FluentD proxy we have (" + }, + { + "type": "link", + "url": "https://github.com/OpenLineage/OpenLineage/tree/main/proxy/fluentd" + }, + { + "type": "text", + "text": ") to copy event stream (alerting is just one of usecases for lineage events) and write fluentd plugin to send it asynchronously further to alerting service like PagerDuty.\n\nIt looks cool to me but I never had enough time to test this approach." + } + ] + } + ] + } + ], + "team": "T01CWUYP5AR", + "thread_ts": "1696350897.139129", + "parent_user_id": "U03D8K119LJ", + "reactions": [ + { + "name": "+1", + "users": [ + "U03D8K119LJ" + ], + "count": 1 + } + ] + } + ] + }, + { + "client_msg_id": "5916a03e-8696-4c6d-ab94-103622360360", + "type": "message", + "text": "\n*We released OpenLineage 1.3.1!*\n*Added:*\n• Airflow: add some basic stats to the Airflow integration `#1845` \n• Airflow: add columns as schema facet for `airflow.lineage.Table` (if defined) `#2138` \n• DBT: add SQLSERVER to supported dbt profile types `#2136` \n• Spark: support for latest 3.5 `#2118` \n*Fixed:*\n• Airflow: fix find-links path in tox `#2139` \n• Airflow: add more graceful logging when no OpenLineage provider installed `#2141` \n• Spark: fix bug in PathUtils’ `prepareDatasetIdentifierFromDefaultTablePath` (CatalogTable) to correctly preserve scheme from `CatalogTable`’s location `#2142` \nThanks to all the contributors, including new contributor <@U05TZE47F2S>!\n*Release:* \n*Changelog:* \n*Commit history:* \n*Maven:* \n*PyPI:* ", + "user": "U02LXF3HUN7", + "ts": "1696344963.496819", + "blocks": [ + { + "type": "rich_text", + "block_id": "0+5XB", + "elements": [ + { + "type": "rich_text_section", + "elements": [ + { + "type": "broadcast", + "range": "channel" + }, + { + "type": "text", + "text": "\n" + }, + { + "type": "text", + "text": "We released OpenLineage 1.3.1!", + "style": { + "bold": true + } + }, + { + "type": "text", + "text": "\n" + }, + { + "type": "text", + "text": "Added:", + "style": { + "bold": true + } + }, + { + "type": "text", + "text": "\n" + } + ] + }, + { + "type": "rich_text_list", + "elements": [ + { + "type": "rich_text_section", + "elements": [ + { + "type": "text", + "text": "Airflow: add some basic stats to the Airflow integration " + }, + { + "type": "text", + "text": "#1845", + "style": { + "code": true + } + }, + { + "type": "text", + "text": " " + }, + { + "type": "link", + "url": "https://github.com/harels", + "text": "@harels", + "unsafe": true + } + ] + }, + { + "type": "rich_text_section", + "elements": [ + { + "type": "text", + "text": "Airflow: add columns as schema facet for " + }, + { + "type": "text", + "text": "airflow.lineage.Table", + "style": { + "code": true + } + }, + { + "type": "text", + "text": " (if defined) " + }, + { + "type": "text", + "text": "#2138", + "style": { + "code": true + } + }, + { + "type": "text", + "text": " " + }, + { + "type": "link", + "url": "https://github.com/erikalfthan", + "text": "@erikalfthan", + "unsafe": true + } + ] + }, + { + "type": "rich_text_section", + "elements": [ + { + "type": "text", + "text": "DBT: add SQLSERVER to supported dbt profile types " + }, + { + "type": "text", + "text": "#2136", + "style": { + "code": true + } + }, + { + "type": "text", + "text": " " + }, + { + "type": "link", + "url": "https://github.com/erikalfthan", + "text": "@erikalfthan", + "unsafe": true + } + ] + }, + { + "type": "rich_text_section", + "elements": [ + { + "type": "text", + "text": "Spark: support for latest 3.5 " + }, + { + "type": "text", + "text": "#2118", + "style": { + "code": true + } + }, + { + "type": "text", + "text": " " + }, + { + "type": "link", + "url": "https://github.com/pawel-big-lebowski", + "text": "@pawel-big-lebowski", + "unsafe": true + } + ] + } + ], + "style": "bullet", + "indent": 0, + "border": 0 + }, + { + "type": "rich_text_section", + "elements": [ + { + "type": "text", + "text": "Fixed:", + "style": { + "bold": true + } + }, + { + "type": "text", + "text": "\n" + } + ] + }, + { + "type": "rich_text_list", + "elements": [ + { + "type": "rich_text_section", + "elements": [ + { + "type": "text", + "text": "Airflow: fix find-links path in tox " + }, + { + "type": "text", + "text": "#2139", + "style": { + "code": true + } + }, + { + "type": "text", + "text": " " + }, + { + "type": "link", + "url": "https://github.com/JDarDagran", + "text": "@JDarDagran", + "unsafe": true + } + ] + }, + { + "type": "rich_text_section", + "elements": [ + { + "type": "text", + "text": "Airflow: add more graceful logging when no OpenLineage provider installed " + }, + { + "type": "text", + "text": "#2141", + "style": { + "code": true + } + }, + { + "type": "text", + "text": " " + }, + { + "type": "link", + "url": "https://github.com/JDarDagran", + "text": "@JDarDagran", + "unsafe": true + } + ] + }, + { + "type": "rich_text_section", + "elements": [ + { + "type": "text", + "text": "Spark: fix bug in PathUtils’ " + }, + { + "type": "text", + "text": "prepareDatasetIdentifierFromDefaultTablePath", + "style": { + "code": true + } + }, + { + "type": "text", + "text": " (CatalogTable) to correctly preserve scheme from " + }, + { + "type": "text", + "text": "CatalogTable", + "style": { + "code": true + } + }, + { + "type": "text", + "text": "’s location " + }, + { + "type": "text", + "text": "#2142", + "style": { + "code": true + } + }, + { + "type": "text", + "text": " " + }, + { + "type": "link", + "url": "https://github.com/d-m-h", + "text": "@d-m-h", + "unsafe": true + } + ] + } + ], + "style": "bullet", + "indent": 0, + "border": 0 + }, + { + "type": "rich_text_section", + "elements": [ + { + "type": "text", + "text": "Thanks to all the contributors, including new contributor " + }, + { + "type": "user", + "user_id": "U05TZE47F2S" + }, + { + "type": "text", + "text": "!\n" + }, + { + "type": "text", + "text": "Release:", + "style": { + "bold": true + } + }, + { + "type": "text", + "text": " " + }, + { + "type": "link", + "url": "https://github.com/OpenLineage/OpenLineage/releases/tag/1.3.1" + }, + { + "type": "text", + "text": "\n" + }, + { + "type": "text", + "text": "Changelog: ", + "style": { + "bold": true + } + }, + { + "type": "link", + "url": "https://github.com/OpenLineage/OpenLineage/blob/main/CHANGELOG.md" + }, + { + "type": "text", + "text": "\n" + }, + { + "type": "text", + "text": "Commit history:", + "style": { + "bold": true + } + }, + { + "type": "text", + "text": " " + }, + { + "type": "link", + "url": "https://github.com/OpenLineage/OpenLineage/compare/1.2.2...1.3.1" + }, + { + "type": "text", + "text": "\n" + }, + { + "type": "text", + "text": "Maven:", + "style": { + "bold": true + } + }, + { + "type": "text", + "text": " " + }, + { + "type": "link", + "url": "https://oss.sonatype.org/#nexus-search;quick~openlineage" + }, + { + "type": "text", + "text": "\n" + }, + { + "type": "text", + "text": "PyPI:", + "style": { + "bold": true + } + }, + { + "type": "text", + "text": " " + }, + { + "type": "link", + "url": "https://pypi.org/project/openlineage-python/" + } + ] + } + ] + } + ], + "team": "T01CWUYP5AR", + "edited": { + "user": "U02LXF3HUN7", + "ts": "1696346004.000000" + }, + "thread_ts": "1696344963.496819", + "reply_count": 1, + "reply_users_count": 1, + "latest_reply": "1696419779.225109", + "reply_users": [ + "U01HVNU6A4C" + ], + "is_locked": false, + "subscribed": true, + "last_read": "1696419779.225109", + "reactions": [ + { + "name": "+1", + "users": [ + "U05T8BJD4DU", + "U05KKM07PJP", + "U05QA2D1XNV", + "U01HVNU6A4C" + ], + "count": 4 + }, + { + "name": "tada", + "users": [ + "U0323HG8C8H" + ], + "count": 1 + } + ], + "replies": [ + { + "client_msg_id": "08089f4a-1e26-4692-91ee-b81c4e791143", + "type": "message", + "text": "Any chance we can do a 1.3.2 soonish to include instead of waiting for the next monthly release?", + "user": "U01HVNU6A4C", + "ts": "1696419779.225109", + "blocks": [ + { + "type": "rich_text", + "block_id": "qcdyH", + "elements": [ + { + "type": "rich_text_section", + "elements": [ + { + "type": "text", + "text": "Any chance we can do a 1.3.2 soonish to include " + }, + { + "type": "link", + "url": "https://github.com/OpenLineage/OpenLineage/pull/2151" + }, + { + "type": "text", + "text": " instead of waiting for the next monthly release?" + } + ] + } + ] + } + ], + "team": "T01CWUYP5AR", + "attachments": [ + { + "id": 1, + "footer_icon": "https://slack.github.com/static/img/favicon-neutral.png", + "ts": 1696337382, + "color": "6f42c1", + "bot_id": "B01VA0FB340", + "app_unfurl_url": "https://github.com/OpenLineage/OpenLineage/pull/2151", + "is_app_unfurl": true, + "app_id": "A01BP7R4KNY", + "fallback": "#2151 Allow setting client's endpoint via environment variable", + "text": "*Problem*\n\nCurrently, it's not possible to set the OpenLineage endpoint (hard-coded to `/api/v1/lineage`) using an environment variable when running the Airflow integration.\n\n*Solution*\n\nGiven that it's not possible to create the client manually in Airflow, especially now that OpenLineage has become an official Airflow provider, this change seems like the only feasible solution.\n\n☐ Your change modifies the OpenLineage model\n☐ Your change modifies one or more OpenLineage \n\n*One-line summary:*\n\nAllow setting client's endpoint via environment variable.\n\n*Checklist*\n\n☑︎ You've your work\n☑︎ Your pull request title follows our \n☑︎ Your changes are accompanied by tests (_if relevant_)\n☑︎ Your change contains a and is self-contained\n☑︎ You've updated any relevant documentation (_if relevant_)\n☑︎ Your comment includes a one-liner for the changelog about the specific purpose of the change (_if necessary_)\n☐ You've versioned the core OpenLineage model or facets according to (_if relevant_)\n☐ You've added a to source files (_if relevant_)\n\n* * *\n\nSPDX-License-Identifier: Apache-2.0 \nCopyright 2018-2023 contributors to the OpenLineage project", + "title": "#2151 Allow setting client's endpoint via environment variable", + "title_link": "https://github.com/OpenLineage/OpenLineage/pull/2151", + "footer": "", + "fields": [ + { + "value": "documentation, client/python", + "title": "Labels", + "short": true + }, + { + "value": "4", + "title": "Comments", + "short": true + } + ], + "mrkdwn_in": [ + "text" + ] + } + ], + "thread_ts": "1696344963.496819", + "parent_user_id": "U02LXF3HUN7" + } + ] + }, + { + "client_msg_id": "675885a1-9307-4628-842d-94abb653b406", + "type": "message", + "text": "Hi folks - I'm wondering if its just me, but does `io.openlineage:openlineage-sql-java:1.2.2` ship with the `arm64.dylib` binary? When i try and run code that uses the Java package on an Apple M1, the binary isn't found, The workaround is to checkout 1.2.2 and then build and publish it locally.", + "user": "U05FLJE4GDU", + "ts": "1696319076.770719", + "blocks": [ + { + "type": "rich_text", + "block_id": "n5OHe", + "elements": [ + { + "type": "rich_text_section", + "elements": [ + { + "type": "text", + "text": "Hi folks - I'm wondering if its just me, but does " + }, + { + "type": "text", + "text": "io.openlineage:openlineage-sql-java:1.2.2", + "style": { + "code": true + } + }, + { + "type": "text", + "text": " ship with the " + }, + { + "type": "text", + "text": "arm64.dylib", + "style": { + "code": true + } + }, + { + "type": "text", + "text": " binary? When i try and run code that uses the Java package on an Apple M1, the binary isn't found, The workaround is to checkout 1.2.2 and then build and publish it locally." + } + ] + } + ] + } + ], + "team": "T01CWUYP5AR", + "thread_ts": "1696319076.770719", + "reply_count": 5, + "reply_users_count": 3, + "latest_reply": "1698076572.116649", + "reply_users": [ + "U02MK6YNAQ5", + "U05FLJE4GDU", + "U01RA9B5GG2" + ], + "is_locked": false, + "subscribed": false, + "replies": [ + { + "client_msg_id": "ab23da6a-0e7b-4a69-ae45-674463cffd7c", + "type": "message", + "text": "Not sure if I follow your question. Whenever OL is released, there is a script new-version.sh - being run and modify the codebase.\n\nSo, If you pull the code, it contains OL version that has not been released yet and in case of dependencies, one need to build them on their own.\n\nFor example, here Preparation section describes how to build openlineage-java and openlineage-sql in order to build openlineage-spark.", + "user": "U02MK6YNAQ5", + "ts": "1696338098.877059", + "blocks": [ + { + "type": "rich_text", + "block_id": "7ZhDU", + "elements": [ + { + "type": "rich_text_section", + "elements": [ + { + "type": "text", + "text": "Not sure if I follow your question. Whenever OL is released, there is a script new-version.sh - " + }, + { + "type": "link", + "url": "https://github.com/OpenLineage/OpenLineage/blob/main/new-version.sh" + }, + { + "type": "text", + "text": " being run and modify the codebase.\n\nSo, If you pull the code, it contains OL version that has not been released yet and in case of dependencies, one need to build them on their own.\n\nFor example, here " + }, + { + "type": "link", + "url": "https://github.com/OpenLineage/OpenLineage/tree/main/integration/spark#preparation" + }, + { + "type": "text", + "text": " Preparation section describes how to build openlineage-java and openlineage-sql in order to build openlineage-spark." + } + ] + } + ] + } + ], + "team": "T01CWUYP5AR", + "attachments": [ + { + "id": 1, + "footer_icon": "https://slack.github.com/static/img/favicon-neutral.png", + "color": "24292f", + "bot_id": "B01VA0FB340", + "app_unfurl_url": "https://github.com/OpenLineage/OpenLineage/blob/main/new-version.sh", + "is_app_unfurl": true, + "app_id": "A01BP7R4KNY", + "fallback": "", + "title": "", + "footer": "", + "mrkdwn_in": [ + "text" + ] + } + ], + "thread_ts": "1696319076.770719", + "parent_user_id": "U05FLJE4GDU" + }, + { + "client_msg_id": "ee51bf80-f877-41b6-87e7-9ba830e97872", + "type": "message", + "text": "Hmm. Let's elaborate my use case a bit.\n\nWe run Apache Hive on-premise. Hive provides query execution hooks for pre-query, post-query, and I *think* failed query.\n\nAny way, as part of the hook, you're given the query string.\n\nSo I, naturally, tried to pass the query string into `OpenLineageSql.parse(Collections.singletonList(hookContext.getQueryPlan().getQueryStr()), \"hive\")` in order to test this out.\n\nI was using `openlineage-sql-java:1.2.2` at that time, and no matter what query string I gave it, *nothing* was returned.\n\nI then stepped through the code and noticed that it was looking for the `arm64` lib, and I noticed that that package (downloaded from maven central) lacked that particular native binary.", + "user": "U05FLJE4GDU", + "ts": "1696411646.822449", + "blocks": [ + { + "type": "rich_text", + "block_id": "BdeNT", + "elements": [ + { + "type": "rich_text_section", + "elements": [ + { + "type": "text", + "text": "Hmm. Let's elaborate my use case a bit.\n\nWe run Apache Hive on-premise. Hive provides query execution hooks for pre-query, post-query, and I " + }, + { + "type": "text", + "text": "think", + "style": { + "bold": true + } + }, + { + "type": "text", + "text": " failed query.\n\nAny way, as part of the hook, you're given the query string.\n\nSo I, naturally, tried to pass the query string into " + }, + { + "type": "text", + "text": "OpenLineageSql.parse(Collections.singletonList(hookContext.getQueryPlan().getQueryStr()), \"hive\")", + "style": { + "code": true + } + }, + { + "type": "text", + "text": " in order to test this out.\n\nI was using " + }, + { + "type": "text", + "text": "openlineage-sql-java:1.2.2", + "style": { + "code": true + } + }, + { + "type": "text", + "text": " at that time, and no matter what query string I gave it, " + }, + { + "type": "text", + "text": "nothing", + "style": { + "bold": true + } + }, + { + "type": "text", + "text": " was returned.\n\nI then stepped through the code and noticed that it was looking for the " + }, + { + "type": "text", + "text": "arm64", + "style": { + "code": true + } + }, + { + "type": "text", + "text": " lib, and I noticed that that package (downloaded from maven central) lacked that particular native binary." + } + ] + } + ] + } + ], + "team": "T01CWUYP5AR", + "edited": { + "user": "U05FLJE4GDU", + "ts": "1696411650.000000" + }, + "thread_ts": "1696319076.770719", + "parent_user_id": "U05FLJE4GDU" + }, + { + "client_msg_id": "be4bbc1f-2209-4b5a-97e2-60b48eaac361", + "type": "message", + "text": "I hope that helps.", + "user": "U05FLJE4GDU", + "ts": "1696411656.832229", + "blocks": [ + { + "type": "rich_text", + "block_id": "R3Bea", + "elements": [ + { + "type": "rich_text_section", + "elements": [ + { + "type": "text", + "text": "I hope that helps." + } + ] + } + ] + } + ], + "team": "T01CWUYP5AR", + "thread_ts": "1696319076.770719", + "parent_user_id": "U05FLJE4GDU", + "reactions": [ + { + "name": "+1", + "users": [ + "U02MK6YNAQ5" + ], + "count": 1 + } + ] + }, + { + "client_msg_id": "dfc061b2-4bba-4361-8d9f-6aef9649904b", + "type": "message", + "text": "I get in now. In Circle CI we do have 3 build steps:\n``` - build-integration-sql-x86\n - build-integration-sql-arm\n - build-integration-sql-macos```\nbut no mac m1. I think at that time circle CI did not have a proper resource class in free plan. Additionally, <@U01RA9B5GG2> would prefer to migrate this to github actions as he claims this can be achieved there in a cleaner way ().\n\nFeel free to create an issue for this. Others would be able to upvote it in case they have similar experience.", + "user": "U02MK6YNAQ5", + "ts": "1696424582.779769", + "blocks": [ + { + "type": "rich_text", + "block_id": "a95tz", + "elements": [ + { + "type": "rich_text_section", + "elements": [ + { + "type": "text", + "text": "I get in now. In Circle CI we do have 3 build steps:\n" + } + ] + }, + { + "type": "rich_text_preformatted", + "elements": [ + { + "type": "text", + "text": " - build-integration-sql-x86\n - build-integration-sql-arm\n - build-integration-sql-macos" + } + ], + "border": 0 + }, + { + "type": "rich_text_section", + "elements": [ + { + "type": "text", + "text": "but no mac m1. I think at that time circle CI did not have a proper resource class in free plan. Additionally, " + }, + { + "type": "user", + "user_id": "U01RA9B5GG2" + }, + { + "type": "text", + "text": " would prefer to migrate this to github actions as he claims this can be achieved there in a cleaner way (" + }, + { + "type": "link", + "url": "https://github.com/OpenLineage/OpenLineage/issues/1624" + }, + { + "type": "text", + "text": ").\n\nFeel free to create an issue for this. Others would be able to upvote it in case they have similar experience." + } + ] + } + ] + } + ], + "team": "T01CWUYP5AR", + "attachments": [ + { + "id": 1, + "footer_icon": "https://slack.github.com/static/img/favicon-neutral.png", + "ts": 1676307633, + "color": "36a64f", + "bot_id": "B01VA0FB340", + "app_unfurl_url": "https://github.com/OpenLineage/OpenLineage/issues/1624", + "is_app_unfurl": true, + "app_id": "A01BP7R4KNY", + "fallback": "#1624 CI: PoC building openlineage-sql on GitHub Actions with actions-rs", + "text": "Building sql parser currently is the most convoluted CI process due to need to construct different binaries in multiple dimensions; both for Java and Python, and for multiple architectures; like Linux x86, Linux ARM, MacOS x86 etc. The jobs also differ in different context: release workflow has different jobs than build one, which in turn does not build all of the architectures.\n\nTo simplify that, we should try using GitHub Actions with that should solve the problems we've currently had to replicate manually.\n\nEnd result of that task should be having various SQL artifacts produced by GitHub actions and available by GH Actions artifacts API:", + "title": "#1624 CI: PoC building openlineage-sql on GitHub Actions with actions-rs", + "title_link": "https://github.com/OpenLineage/OpenLineage/issues/1624", + "footer": "", + "fields": [ + { + "value": "", + "title": "Assignees", + "short": true + }, + { + "value": "ci, integration/sql", + "title": "Labels", + "short": true + } + ], + "mrkdwn_in": [ + "text" + ] + } + ], + "thread_ts": "1696319076.770719", + "parent_user_id": "U05FLJE4GDU" + }, + { + "client_msg_id": "35f6d93f-4aa6-4076-8694-f9320acdba2b", + "type": "message", + "text": "It doesn't have the free resource class still :disappointed:\nWe're blocked on that unfortunately. Other solution would be to migrate to GH actions, where most of our solution could be replaced by something like that ", + "user": "U01RA9B5GG2", + "ts": "1698076572.116649", + "blocks": [ + { + "type": "rich_text", + "block_id": "/SjVu", + "elements": [ + { + "type": "rich_text_section", + "elements": [ + { + "type": "text", + "text": "It doesn't have the free resource class still " + }, + { + "type": "emoji", + "name": "disappointed", + "unicode": "1f61e" + }, + { + "type": "text", + "text": "\nWe're blocked on that unfortunately. Other solution would be to migrate to GH actions, where most of our solution could be replaced by something like that " + }, + { + "type": "link", + "url": "https://github.com/PyO3/maturin-action" + } + ] + } + ] + } + ], + "team": "T01CWUYP5AR", + "attachments": [ + { + "id": 1, + "color": "24292f", + "bot_id": "B01VA0FB340", + "app_unfurl_url": "https://github.com/PyO3/maturin-action", + "is_app_unfurl": true, + "app_id": "A01BP7R4KNY", + "fallback": "PyO3/maturin-action", + "text": "GitHub Action to install and run a custom maturin command with built-in support for cross compilation", + "title": "PyO3/maturin-action", + "fields": [ + { + "value": "98", + "title": "Stars", + "short": true + }, + { + "value": "TypeScript", + "title": "Language", + "short": true + } + ] + } + ], + "thread_ts": "1696319076.770719", + "parent_user_id": "U05FLJE4GDU" + } + ] + }, + { + "client_msg_id": "9be7823a-a74a-457e-a605-351e82874e29", + "type": "message", + "text": "\nThe September issue of is here! This issue covers the big news about OpenLineage coming out of Airflow Summit, progress on the Airflow Provider, highlights from our meetup in Toronto, and much more.\nTo get the newsletter directly in your inbox each month, sign up .", + "user": "U02LXF3HUN7", + "ts": "1696264108.497989", + "blocks": [ + { + "type": "rich_text", + "block_id": "uK1Ve", + "elements": [ + { + "type": "rich_text_section", + "elements": [ + { + "type": "broadcast", + "range": "channel" + }, + { + "type": "text", + "text": "\nThe September issue of " + }, + { + "type": "link", + "url": "https://mailchi.mp/3b9bbb3eba23/openlineage-news-july-9591485?e=ce16eef4ef", + "text": "OpenLineage News" + }, + { + "type": "text", + "text": " is here! This issue covers the big news about OpenLineage coming out of Airflow Summit, progress on the Airflow Provider, highlights from our meetup in Toronto, and much more.\nTo get the newsletter directly in your inbox each month, sign up " + }, + { + "type": "link", + "url": "http://bit.ly/OL_news", + "text": "here" + }, + { + "type": "text", + "text": "." + } + ] + } + ] + } + ], + "team": "T01CWUYP5AR", + "attachments": [ + { + "from_url": "http://bit.ly/OL_news", + "id": 1, + "original_url": "http://bit.ly/OL_news", + "fallback": "OpenLineage Project", + "text": "OpenLineage Project Email Forms", + "title": "OpenLineage Project", + "title_link": "http://bit.ly/OL_news", + "service_name": "apache.us14.list-manage.com" + } + ], + "reactions": [ + { + "name": "duck", + "users": [ + "U01HNKK4XAM", + "U02MK6YNAQ5" + ], + "count": 2 + }, + { + "name": "fire", + "users": [ + "U01DCMDFHBK", + "U02S6F54MAB", + "U02MK6YNAQ5" + ], + "count": 3 + } + ] + }, + { + "client_msg_id": "a754cd47-0e9f-4a33-b417-c0dc64ccfe5e", + "type": "message", + "text": "\nHello all, I’d like to open a vote to release OpenLineage 1.3.0, including:\n• support for Spark 3.5 in the Spark integration\n• scheme preservation bug fix in the Spark integration\n• find-links path in tox bug in the Airflow integration fix\n• more graceful logging when no OL provider is installed in the Airflow integration\n• columns as schema facet for airflow.lineage.Table addition\n• SQLSERVER to supported dbt profile types addition\nThree +1s from committers will authorize. Thanks in advance.", + "user": "U02LXF3HUN7", + "ts": "1696262312.791719", + "blocks": [ + { + "type": "rich_text", + "block_id": "QqIB1", + "elements": [ + { + "type": "rich_text_section", + "elements": [ + { + "type": "broadcast", + "range": "channel" + }, + { + "type": "text", + "text": "\nHello all, I’d like to open a vote to release OpenLineage 1.3.0, including:\n" + } + ] + }, + { + "type": "rich_text_list", + "elements": [ + { + "type": "rich_text_section", + "elements": [ + { + "type": "text", + "text": "support for Spark 3.5 in the Spark integration" + } + ] + }, + { + "type": "rich_text_section", + "elements": [ + { + "type": "text", + "text": "scheme preservation bug fix in the Spark integration" + } + ] + }, + { + "type": "rich_text_section", + "elements": [ + { + "type": "text", + "text": "find-links path in tox bug in the Airflow integration fix" + } + ] + }, + { + "type": "rich_text_section", + "elements": [ + { + "type": "text", + "text": "more graceful logging when no OL provider is installed in the Airflow integration" + } + ] + }, + { + "type": "rich_text_section", + "elements": [ + { + "type": "text", + "text": "columns as schema facet for airflow.lineage.Table addition" + } + ] + }, + { + "type": "rich_text_section", + "elements": [ + { + "type": "text", + "text": "SQLSERVER to supported dbt profile types addition" + } + ] + } + ], + "style": "bullet", + "indent": 0, + "border": 0 + }, + { + "type": "rich_text_section", + "elements": [ + { + "type": "text", + "text": "Three +1s from committers will authorize. Thanks in advance." + } + ] + } + ] + } + ], + "team": "T01CWUYP5AR", + "thread_ts": "1696262312.791719", + "reply_count": 12, + "reply_users_count": 3, + "latest_reply": "1696532708.916989", + "reply_users": [ + "U02LXF3HUN7", + "U05T8BJD4DU", + "U01HNKK4XAM" + ], + "is_locked": false, + "subscribed": true, + "last_read": "1696532708.916989", + "reactions": [ + { + "name": "raised_hands", + "users": [ + "U01HNKK4XAM", + "U02MK6YNAQ5", + "U05TU0U224A" + ], + "count": 3 + }, + { + "name": "+1", + "users": [ + "U05T8BJD4DU", + "U02MK6YNAQ5" + ], + "count": 2 + }, + { + "name": "heavy_plus_sign", + "users": [ + "U01DCMDFHBK", + "U02S6F54MAB", + "U05TZE47F2S", + "U01DCLP0GU9" + ], + "count": 4 + } + ], + "replies": [ + { + "client_msg_id": "3c57a2da-2f2f-4456-8dbb-d5e5b6116ea4", + "type": "message", + "text": "Thanks all. The release is authorized and will be initiated within 2 business days.", + "user": "U02LXF3HUN7", + "ts": "1696280408.812339", + "blocks": [ + { + "type": "rich_text", + "block_id": "nrzDe", + "elements": [ + { + "type": "rich_text_section", + "elements": [ + { + "type": "text", + "text": "Thanks all. The release is authorized and will be initiated within 2 business days." + } + ] + } + ] + } + ], + "team": "T01CWUYP5AR", + "thread_ts": "1696262312.791719", + "parent_user_id": "U02LXF3HUN7" + }, + { + "client_msg_id": "f1310bc9-9c1a-4e4a-83a1-17da51247481", + "type": "message", + "text": "looking forward to that, I am seeing inconsistent results in Databricks for Spark 3.4+, sometimes there's no inputs / outputs, hope that is fixed?", + "user": "U05T8BJD4DU", + "ts": "1696281106.654189", + "blocks": [ + { + "type": "rich_text", + "block_id": "O2lR9", + "elements": [ + { + "type": "rich_text_section", + "elements": [ + { + "type": "text", + "text": "looking forward to that, I am seeing inconsistent results in Databricks for Spark 3.4+, sometimes there's no inputs / outputs, hope that is fixed?" + } + ] + } + ] + } + ], + "team": "T01CWUYP5AR", + "thread_ts": "1696262312.791719", + "parent_user_id": "U02LXF3HUN7" + }, + { + "client_msg_id": "5565759c-a9c0-4bbf-8f2a-bb64e1d26903", + "type": "message", + "text": "<@U05T8BJD4DU> if it isn’t fixed for you, would love it if you could open up an issue that will allow us to reproduce and fix", + "user": "U01HNKK4XAM", + "ts": "1696341564.380589", + "blocks": [ + { + "type": "rich_text", + "block_id": "LbQzV", + "elements": [ + { + "type": "rich_text_section", + "elements": [ + { + "type": "user", + "user_id": "U05T8BJD4DU" + }, + { + "type": "text", + "text": " if it isn’t fixed for you, would love it if you could open up an issue that will allow us to reproduce and fix" + } + ] + } + ] + } + ], + "team": "T01CWUYP5AR", + "thread_ts": "1696262312.791719", + "parent_user_id": "U02LXF3HUN7", + "reactions": [ + { + "name": "+1", + "users": [ + "U05T8BJD4DU" + ], + "count": 1 + } + ] + }, + { + "client_msg_id": "b95f84ae-a245-4be9-ad55-43fd85f18bc2", + "type": "message", + "text": "<@U01HNKK4XAM> the issue still exists -> Spark 3.4 and above, including 3.5, saveAsTable and create table won't have inputs and outputs in Databricks", + "user": "U05T8BJD4DU", + "ts": "1696379020.978749", + "blocks": [ + { + "type": "rich_text", + "block_id": "KFFSy", + "elements": [ + { + "type": "rich_text_section", + "elements": [ + { + "type": "user", + "user_id": "U01HNKK4XAM" + }, + { + "type": "text", + "text": " the issue still exists -> Spark 3.4 and above, including 3.5, saveAsTable and create table won't have inputs and outputs in Databricks" + } + ] + } + ] + } + ], + "team": "T01CWUYP5AR", + "thread_ts": "1696262312.791719", + "parent_user_id": "U02LXF3HUN7" + }, + { + "client_msg_id": "1e2d90d3-c11b-405a-9f0d-35a107543a29", + "type": "message", + "text": "", + "user": "U05T8BJD4DU", + "ts": "1696379415.171809", + "blocks": [ + { + "type": "rich_text", + "block_id": "vmCqe", + "elements": [ + { + "type": "rich_text_section", + "elements": [ + { + "type": "link", + "url": "https://github.com/OpenLineage/OpenLineage/issues/2124" + } + ] + } + ] + } + ], + "team": "T01CWUYP5AR", + "attachments": [ + { + "id": 1, + "footer_icon": "https://slack.github.com/static/img/favicon-neutral.png", + "ts": 1695498902, + "color": "36a64f", + "bot_id": "B01VA0FB340", + "app_unfurl_url": "https://github.com/OpenLineage/OpenLineage/issues/2124", + "is_app_unfurl": true, + "app_id": "A01BP7R4KNY", + "fallback": "#2124 Same Delta Table not catching the location on write", + "text": "*What is the target system?*\n\nSpark / Databricks\n\n*What kind of integration is this?*\n\n☐ Produces OpenLineage metadata\n☐ Consumes OpenLineage metadata\n☐ Something else\n\n*How should this integration be implemented?*\n\nI am using OL 1.2.2, Azure Databricks Runtime 11.3 LTS. When creating a table writing into a ADLS location, OL won't be able to catch the location of the output. But when I read the same object it will be able to read the location as INPUT.\n\nPlease note I have also tested Databricks Runtime 13.3 LTS, Spark 3.4.1 - it will give correct ADLS location in INPUT but the input will only show up once in a blue moon. Most of the time the inputs and outputs are blank.\n\n```\n \"inputs\": [],\n \"outputs\": []\n```\n\n```\nCREATE OR REPLACE TABLE transactions_adj\nUSING DELTA LOCATION ''\nAS\n SELECT\n household_id,\n basket_id,\n week_no,\n day,\n transaction_time,\n store_id,\n product_id,\n amount_list,\n campaign_coupon_discount,\n manuf_coupon_discount,\n manuf_coupon_match_discount,\n total_coupon_discount,\n instore_discount,\n amount_paid,\n units\n FROM (\n SELECT \n household_id,\n basket_id,\n week_no,\n day,\n transaction_time,\n store_id,\n product_id,\n COALESCE(sales_amount - discount_amount - coupon_discount - coupon_discount_match,0.0) as amount_list,\n CASE \n WHEN COALESCE(coupon_discount_match,0.0) = 0.0 THEN -1 * COALESCE(coupon_discount,0.0) \n ELSE 0.0 \n END as campaign_coupon_discount,\n CASE \n WHEN COALESCE(coupon_discount_match,0.0) != 0.0 THEN -1 * COALESCE(coupon_discount,0.0) \n ELSE 0.0 \n END as manuf_coupon_discount,\n -1 * COALESCE(coupon_discount_match,0.0) as manuf_coupon_match_discount,\n -1 * COALESCE(coupon_discount - coupon_discount_match,0.0) as total_coupon_discount,\n COALESCE(-1 * discount_amount,0.0) as instore_discount,\n COALESCE(sales_amount,0.0) as `amount_paid,`\n quantity as units\n FROM transactions\n );\n```\n\nHere's the COMPLETE event:\n\n```\n\n \"outputs\":[\n {\n \"namespace\":\"dbfs\",\n \"name\":\"/user/hive/warehouse/journey.db/transactions_adj\",\n \"facets\":{\n \"dataSource\":{\n \"_producer\":\"\",\n \"_schemaURL\":\"\",\n \"name\":\"dbfs\",\n \"uri\":\"dbfs\"\n },\n\n```\n\nBelow logical plan shows the path:\n\n```\n== Analyzed Logical Plan ==\nnum_affected_rows: bigint, num_inserted_rows: bigint\nReplaceTableAsSelect TableSpec(Map(),Some(DELTA),Map(),Some(),None,None,false,Set()), true\n:- ResolvedIdentifier com.databricks.sql.managedcatalog.UnityCatalogV2Proxy@6251a8df, default.transactions_adj\n+- Project [household_id#184, basket_id#185L, week_no#193, day#186, transaction_time#192, store_id#190, product_id#187, amount_list#147, campaign_coupon_discount#148, manuf_coupon_discount#149, manuf_coupon_match_discount#150, total_coupon_discount#151, instore_discount#152, amount_paid#153, units#154]\n +- SubqueryAlias __auto_generated_subquery_name\n +- Project [household_id#184, basket_id#185L, week_no#193, day#186, transaction_time#192, store_id#190, product_id#187, coalesce(cast((((sales_amount#189 - discount_amount#191) - coupon_discount#194) - coupon_discount_match#195) as double), cast(0.0 as double)) AS amount_list#147, CASE WHEN (coalesce(cast(coupon_discount_match#195 as double), cast(0.0 as double)) = cast(0.0 as double)) THEN (cast(-1 as double) * coalesce(cast(coupon_discount#194 as double), cast(0.0 as double))) ELSE cast(0.0 as double) END AS campaign_coupon_discount#148, CASE WHEN NOT (coalesce(cast(coupon_discount_match#195 as double), cast(0.0 as double)) = cast(0.0 as double)) THEN (cast(-1 as double) * coalesce(cast(coupon_discount#194 as double), cast(0.0 as double))) ELSE cast(0.0 as double) END AS manuf_coupon_discount#149, (cast(-1 as double) * coalesce(cast(coupon_discount_match#195 as double), cast(0.0 as double))) AS manuf_coupon_match_discount#150, (cast(-1 as double) * coalesce(cast((coupon_discount#194 - coupon_discount_match#195) as double), cast(0.0 as double))) AS total_coupon_discount#151, coalesce(cast((cast(-1 as float) * discount_amount#191) as double), cast(0.0 as double)) AS instore_discount#152, coalesce(cast(sales_amount#189 as double), cast(0.0 as double)) AS amount_paid#153, quantity#188 AS units#154]\n +- SubqueryAlias spark_catalog.default.transactions\n +- Relation spark_catalog.default.transactions[household_id#184,basket_id#185L,day#186,product_id#187,quantity#188,sales_amount#189,store_id#190,discount_amount#191,transaction_time#192,week_no#193,coupon_discount#194,coupon_discount_match#195] parquet\n```\n\n*Where should this integration be implemented?*\n\n☐ In the target system\n☐ In the OpenLineage repo\n☐ Somewhere else\n\n*Do you plan to make this contribution yourself?*\n\n☐ I am interested in doing this work", + "title": "#2124 Same Delta Table not catching the location on write", + "title_link": "https://github.com/OpenLineage/OpenLineage/issues/2124", + "footer": "", + "fields": [ + { + "value": "integration/spark, integration/databricks", + "title": "Labels", + "short": true + }, + { + "value": "1", + "title": "Comments", + "short": true + } + ], + "mrkdwn_in": [ + "text" + ] + } + ], + "thread_ts": "1696262312.791719", + "parent_user_id": "U02LXF3HUN7" + }, + { + "client_msg_id": "0ba6dc08-dd99-4efe-acdb-2a46aab4e711", + "type": "message", + "text": "and of course this issue still exists", + "user": "U05T8BJD4DU", + "ts": "1696379421.415929", + "blocks": [ + { + "type": "rich_text", + "block_id": "CDeCf", + "elements": [ + { + "type": "rich_text_section", + "elements": [ + { + "type": "text", + "text": "and of course this issue still exists" + } + ] + } + ] + } + ], + "team": "T01CWUYP5AR", + "thread_ts": "1696262312.791719", + "parent_user_id": "U02LXF3HUN7" + }, + { + "client_msg_id": "e9527d1c-1b98-4db2-bc7b-8aa2404dc75b", + "type": "message", + "text": "thanks for posting, we’ll continue looking into this.. if you find any clues that might help, please let us know.", + "user": "U01HNKK4XAM", + "ts": "1696383909.178409", + "blocks": [ + { + "type": "rich_text", + "block_id": "YGzDL", + "elements": [ + { + "type": "rich_text_section", + "elements": [ + { + "type": "text", + "text": "thanks for posting, we’ll continue looking into this.. if you find any clues that might help, please let us know." + } + ] + } + ] + } + ], + "team": "T01CWUYP5AR", + "thread_ts": "1696262312.791719", + "parent_user_id": "U02LXF3HUN7" + }, + { + "client_msg_id": "10fc80ad-5ce4-4aca-82ad-7bc54be5ad6b", + "type": "message", + "text": "is there any instructions on how to hook up a debugger to OL?", + "user": "U05T8BJD4DU", + "ts": "1696383987.807139", + "blocks": [ + { + "type": "rich_text", + "block_id": "0nNm5", + "elements": [ + { + "type": "rich_text_section", + "elements": [ + { + "type": "text", + "text": "is there any instructions on how to hook up a debugger to OL?" + } + ] + } + ] + } + ], + "team": "T01CWUYP5AR", + "thread_ts": "1696262312.791719", + "parent_user_id": "U02LXF3HUN7" + }, + { + "client_msg_id": "aa1993fa-e506-477f-8144-0f6787cfc5ee", + "type": "message", + "text": "<@U02MK6YNAQ5> has been working on adding a debug facet, but more suggestions are more than welcome!", + "user": "U01HNKK4XAM", + "ts": "1696424656.962049", + "blocks": [ + { + "type": "rich_text", + "block_id": "ETaAp", + "elements": [ + { + "type": "rich_text_section", + "elements": [ + { + "type": "user", + "user_id": "U02MK6YNAQ5" + }, + { + "type": "text", + "text": " has been working on adding a debug facet, but more suggestions are more than welcome!" + } + ] + } + ] + } + ], + "team": "T01CWUYP5AR", + "thread_ts": "1696262312.791719", + "parent_user_id": "U02LXF3HUN7" + }, + { + "client_msg_id": "18352e7e-1240-42ac-9d1c-e54a08aae259", + "type": "message", + "text": "", + "user": "U01HNKK4XAM", + "ts": "1696424758.746299", + "blocks": [ + { + "type": "rich_text", + "block_id": "bX9AD", + "elements": [ + { + "type": "rich_text_section", + "elements": [ + { + "type": "link", + "url": "https://github.com/OpenLineage/OpenLineage/pull/2147" + } + ] + } + ] + } + ], + "team": "T01CWUYP5AR", + "attachments": [ + { + "id": 1, + "footer_icon": "https://slack.github.com/static/img/favicon-neutral.png", + "ts": 1696250044, + "color": "36a64f", + "bot_id": "B01VA0FB340", + "app_unfurl_url": "https://github.com/OpenLineage/OpenLineage/pull/2147", + "is_app_unfurl": true, + "app_id": "A01BP7R4KNY", + "fallback": "#2147 [SPARK] add debug facet to help resolving Spark integration issues", + "text": "*Problem*\n\nDebugging openlineage-spark problems is tedious job. We would like to have debug facet that will collect automatically meaningful information when enabled.\n\nCloses: \n\n*Solution*\n\n• Create debug facet (more details in ),\n• Facet is disabled by default,\n• Allow enabling it thourgh SparkConf.\n\n> *Note:* All schema changes require discussion. Please for context.\n\n☐ Your change modifies the OpenLineage model\n☐ Your change modifies one or more OpenLineage \n\nIf you're contributing a new integration, please specify the scope of the integration and how/where it has been tested (e.g., Apache Spark integration supports `S3` and `GCS` filesystem operations, tested with AWS EMR).\n\n*One-line summary:*\n*Checklist*\n\n☑︎ You've your work\n☑︎ Your pull request title follows our \n☑︎ Your changes are accompanied by tests (_if relevant_)\n☑︎ Your change contains a and is self-contained\n☐ You've updated any relevant documentation (_if relevant_)\n☐ Your comment includes a one-liner for the changelog about the specific purpose of the change (_if necessary_)\n☐ You've versioned the core OpenLineage model or facets according to (_if relevant_)\n☐ You've added a to source files (_if relevant_)\n\n* * *\n\nSPDX-License-Identifier: Apache-2.0 \nCopyright 2018-2023 contributors to the OpenLineage project", + "title": "#2147 [SPARK] add debug facet to help resolving Spark integration issues", + "title_link": "https://github.com/OpenLineage/OpenLineage/pull/2147", + "footer": "", + "fields": [ + { + "value": "documentation, integration/spark", + "title": "Labels", + "short": true + }, + { + "value": "", + "title": "Assignees", + "short": true + } + ], + "mrkdwn_in": [ + "text" + ] + } + ], + "thread_ts": "1696262312.791719", + "parent_user_id": "U02LXF3HUN7", + "reactions": [ + { + "name": "eyes", + "users": [ + "U02MK6YNAQ5" + ], + "count": 1 + }, + { + "name": "+1", + "users": [ + "U05T8BJD4DU" + ], + "count": 1 + } + ] + }, + { + "client_msg_id": "6f7e2f93-17d7-44b3-9c22-ff76b63fd77e", + "type": "message", + "text": "<@U02MK6YNAQ5> do you have a build for the PR? Appreciated!", + "user": "U05T8BJD4DU", + "ts": "1696490411.609729", + "blocks": [ + { + "type": "rich_text", + "block_id": "5uhEP", + "elements": [ + { + "type": "rich_text_section", + "elements": [ + { + "type": "user", + "user_id": "U02MK6YNAQ5" + }, + { + "type": "text", + "text": " do you have a build for the PR? Appreciated!" + } + ] + } + ] + } + ], + "team": "T01CWUYP5AR", + "thread_ts": "1696262312.791719", + "parent_user_id": "U02LXF3HUN7" + }, + { + "client_msg_id": "0e21ed3c-4dd4-4254-819f-1328d70a2f29", + "type": "message", + "text": "we’ll ask for a release once it’s reviewed and merged", + "user": "U01HNKK4XAM", + "ts": "1696532708.916989", + "blocks": [ + { + "type": "rich_text", + "block_id": "35mCF", + "elements": [ + { + "type": "rich_text_section", + "elements": [ + { + "type": "text", + "text": "we’ll ask for a release once it’s reviewed and merged" + } + ] + } + ] + } + ], + "team": "T01CWUYP5AR", + "thread_ts": "1696262312.791719", + "parent_user_id": "U02LXF3HUN7" + } + ] + }, + { + "client_msg_id": "72d5b68a-2320-47d1-b453-c910e88a5be6", + "type": "message", + "text": "Are you located in the Brussels area or within commutable distance? Interested in attending a meetup between October 16-20? If so, please DM <@U0323HG8C8H> or myself. TIA", + "user": "U02LXF3HUN7", + "ts": "1695932184.205159", + "blocks": [ + { + "type": "rich_text", + "block_id": "EUBUD", + "elements": [ + { + "type": "rich_text_section", + "elements": [ + { + "type": "text", + "text": "Are you located in the Brussels area or within commutable distance? Interested in attending a meetup between October 16-20? If so, please DM " + }, + { + "type": "user", + "user_id": "U0323HG8C8H" + }, + { + "type": "text", + "text": " or myself. TIA" + } + ] + } + ] + } + ], + "team": "T01CWUYP5AR", + "reactions": [ + { + "name": "heart", + "users": [ + "U0323HG8C8H" + ], + "count": 1 + } + ] + }, + { + "client_msg_id": "94c47910-d424-44da-bd18-73646996e77c", + "type": "message", + "text": "Hello community\nFirst time poster - bear with me :)\n\nI am looking to make a minor PR on the airflow integration (fixing github #2130), and the code change is easy enough, but I fail to install the python environment. I have tried the simple ones\n`OpenLineage/integration/airflow > pip install -e .`\n or\n`OpenLineage/integration/airflow > pip install -r dev-requirements.txt`\nbut they both fail on\n`ERROR: No matching distribution found for openlineage-sql==1.3.0`\n\n(which I think is an unreleased version in the git project)\n\nHow would I go about to install the requirements?\n\n//Erik\n\nPS. Sorry for posting this in general if there is a specific integration or contribution channel - I didnt find a better channel", + "user": "U05TZE47F2S", + "ts": "1695883240.832669", + "blocks": [ + { + "type": "rich_text", + "block_id": "jFDc1", + "elements": [ + { + "type": "rich_text_section", + "elements": [ + { + "type": "text", + "text": "Hello community\nFirst time poster - bear with me :)\n\nI am looking to make a minor PR on the airflow integration (fixing github #2130), and the code change is easy enough, but I fail to install the python environment. I have tried the simple ones\n" + }, + { + "type": "text", + "text": "OpenLineage/integration/airflow > pip install -e .", + "style": { + "code": true + } + }, + { + "type": "text", + "text": "\n or\n" + }, + { + "type": "text", + "text": "OpenLineage/integration/airflow > pip install -r dev-requirements.txt", + "style": { + "code": true + } + }, + { + "type": "text", + "text": "\nbut they both fail on\n" + }, + { + "type": "text", + "text": "ERROR: No matching distribution found for openlineage-sql==1.3.0", + "style": { + "code": true + } + }, + { + "type": "text", + "text": "\n\n(which I think is an unreleased version in the git project)\n\nHow would I go about to install the requirements?\n\n//Erik\n\nPS. Sorry for posting this in general if there is a specific integration or contribution channel - I didnt find a better channel" + } + ] + } + ] + } + ], + "team": "T01CWUYP5AR", + "thread_ts": "1695883240.832669", + "reply_count": 36, + "reply_users_count": 3, + "latest_reply": "1695901421.859499", + "reply_users": [ + "U02MK6YNAQ5", + "U05TZE47F2S", + "U02S6F54MAB" + ], + "is_locked": false, + "subscribed": false, + "replies": [ + { + "client_msg_id": "47f70521-07a4-4520-b030-ebc1a81d0d3e", + "type": "message", + "text": "Hi <@U05TZE47F2S>, the channel is totally OK. I am not airflow integration expert, but what it looks to me, you're missing openlineage-sql library, which is a rust library used to extract lineage from sql queries. This is how we do that in circle ci:\n\n\nand subproject page with build instructions: ", + "user": "U02MK6YNAQ5", + "ts": "1695884688.471259", + "blocks": [ + { + "type": "rich_text", + "block_id": "/qKFV", + "elements": [ + { + "type": "rich_text_section", + "elements": [ + { + "type": "text", + "text": "Hi " + }, + { + "type": "user", + "user_id": "U05TZE47F2S" + }, + { + "type": "text", + "text": ", the channel is totally OK. I am not airflow integration expert, but what it looks to me, you're missing openlineage-sql library, which is a rust library used to extract lineage from sql queries. This is how we do that in circle ci:\n" + }, + { + "type": "link", + "url": "https://app.circleci.com/pipelines/github/OpenLineage/OpenLineage/8080/workflows/aba53369-836c-48f5-a2dd-51bc0740a31c/jobs/140113" + }, + { + "type": "text", + "text": "\n\nand subproject page with build instructions: " + }, + { + "type": "link", + "url": "https://github.com/OpenLineage/OpenLineage/tree/main/integration/sql" + } + ] + } + ] + } + ], + "team": "T01CWUYP5AR", + "thread_ts": "1695883240.832669", + "parent_user_id": "U05TZE47F2S" + }, + { + "client_msg_id": "f864f8f2-d620-4863-b71f-ef587e9d955f", + "type": "message", + "text": "Ok, so I go and \"manually\" build the internal dependency so that it becomes available in the pip cache?\n\nI was hoping for something more automagical, but that should work", + "user": "U05TZE47F2S", + "ts": "1695884843.168339", + "blocks": [ + { + "type": "rich_text", + "block_id": "LDHKU", + "elements": [ + { + "type": "rich_text_section", + "elements": [ + { + "type": "text", + "text": "Ok, so I go and \"manually\" build the internal dependency so that it becomes available in the pip cache?\n\nI was hoping for something more automagical, but that should work" + } + ] + } + ] + } + ], + "team": "T01CWUYP5AR", + "thread_ts": "1695883240.832669", + "parent_user_id": "U05TZE47F2S" + }, + { + "client_msg_id": "335b4e75-5126-48ac-8260-5ead31fa5850", + "type": "message", + "text": "I think so. <@U02S6F54MAB> am I right?", + "user": "U02MK6YNAQ5", + "ts": "1695884886.561369", + "blocks": [ + { + "type": "rich_text", + "block_id": "02eTI", + "elements": [ + { + "type": "rich_text_section", + "elements": [ + { + "type": "text", + "text": "I think so. " + }, + { + "type": "user", + "user_id": "U02S6F54MAB" + }, + { + "type": "text", + "text": " am I right?" + } + ] + } + ] + } + ], + "team": "T01CWUYP5AR", + "thread_ts": "1695883240.832669", + "parent_user_id": "U05TZE47F2S" + }, + { + "client_msg_id": "ea8dcd7b-1269-414c-acd9-d2ed0b9870ed", + "type": "message", + "text": "\nthere’s a guide how to setup the dev environment\n\n> Typically, you first need to build `openlineage-sql` locally (see ). After each release you have to repeat this step in order to bump local version of the package.\nThis might be somewhat exposed more in GitHub repository README as well", + "user": "U02S6F54MAB", + "ts": "1695885507.142819", + "blocks": [ + { + "type": "rich_text", + "block_id": "0thOE", + "elements": [ + { + "type": "rich_text_section", + "elements": [ + { + "type": "link", + "url": "https://openlineage.io/docs/development/developing/python/setup" + }, + { + "type": "text", + "text": "\nthere’s a guide how to setup the dev environment\n\n" + } + ] + }, + { + "type": "rich_text_quote", + "elements": [ + { + "type": "text", + "text": "Typically, you first need to build " + }, + { + "type": "text", + "text": "openlineage-sql", + "style": { + "code": true + } + }, + { + "type": "text", + "text": " locally (see " + }, + { + "type": "link", + "url": "https://github.com/OpenLineage/OpenLineage/blob/main/integration/sql/README.md", + "text": "README" + }, + { + "type": "text", + "text": "). After each release you have to repeat this step in order to bump local version of the package." + } + ] + }, + { + "type": "rich_text_section", + "elements": [ + { + "type": "text", + "text": "\nThis might be somewhat exposed more in GitHub repository README as well" + } + ] + } + ] + } + ], + "team": "T01CWUYP5AR", + "thread_ts": "1695883240.832669", + "parent_user_id": "U05TZE47F2S" + }, + { + "client_msg_id": "346618b6-45c3-4dfc-9288-225a583d5d53", + "type": "message", + "text": "It didnt find the wheel in the cache, but if I used the line in the sql/README.md\n`pip install openlineage-sql --no-index --find-links ../target/wheels --force-reinstall`\nIt is installed and thus skipped/passed when pip later checks if it needs to be installed.\n\nNow I have a second issue because it is expecting me to have mysqlclient-2.2.0 which seems to need a binary\n`Command 'pkg-config --exists mysqlclient' returned non-zero exit status 127`\nand\n`Command 'pkg-config --exists mariadb' returned non-zero exit status 127`\nI am on Ubuntu 22.04 in WSL2. Should I go to apt and grab me a mysql client?", + "user": "U05TZE47F2S", + "ts": "1695886040.154289", + "blocks": [ + { + "type": "rich_text", + "block_id": "FhOUD", + "elements": [ + { + "type": "rich_text_section", + "elements": [ + { + "type": "text", + "text": "It didnt find the wheel in the cache, but if I used the line in the sql/README.md\n" + }, + { + "type": "text", + "text": "pip install openlineage-sql --no-index --find-links ../target/wheels --force-reinstall", + "style": { + "code": true + } + }, + { + "type": "text", + "text": "\nIt is installed and thus skipped/passed when pip later checks if it needs to be installed.\n\nNow I have a second issue because it is expecting me to have mysqlclient-2.2.0 which seems to need a binary\n" + }, + { + "type": "text", + "text": "Command 'pkg-config --exists mysqlclient' returned non-zero exit status 127", + "style": { + "code": true + } + }, + { + "type": "text", + "text": "\nand\n" + }, + { + "type": "text", + "text": "Command 'pkg-config --exists mariadb' returned non-zero exit status 127", + "style": { + "code": true + } + }, + { + "type": "text", + "text": "\nI am on Ubuntu 22.04 in WSL2. Should I go to apt and grab me a mysql client?" + } + ] + } + ] + } + ], + "team": "T01CWUYP5AR", + "thread_ts": "1695883240.832669", + "parent_user_id": "U05TZE47F2S" + }, + { + "client_msg_id": "fe442113-abae-42b0-9b2b-4daefeabf518", + "type": "message", + "text": "> It didnt find the wheel in the cache, but if I used the line in the sql/README.md\n> `pip install openlineage-sql --no-index --find-links ../target/wheels --force-reinstall`\n> It is installed and thus skipped/passed when pip later checks if it needs to be installed.\nThat’s actually expected. You should build new wheel locally and then install it.\n\n> Now I have a second issue because it is expecting me to have mysqlclient-2.2.0 which seems to need a binary\n> `Command 'pkg-config --exists mysqlclient' returned non-zero exit status 127`\n> and\n> `Command 'pkg-config --exists mariadb' returned non-zero exit status 127`\n> I am on Ubuntu 22.04 in WSL2. Should I go to apt and grab me a mysql client?\nWe’ve left some system specific configuration, e.g. mysqlclient, to users as it’s a bit aside from OpenLineage and more of general development task.\n\nprobably\n```sudo apt-get install python3-dev default-libmysqlclient-dev build-essential ```\nshould work", + "user": "U02S6F54MAB", + "ts": "1695886312.730749", + "blocks": [ + { + "type": "rich_text", + "block_id": "M+Muy", + "elements": [ + { + "type": "rich_text_quote", + "elements": [ + { + "type": "text", + "text": "It didnt find the wheel in the cache, but if I used the line in the sql/README.md\n" + }, + { + "type": "text", + "text": "pip install openlineage-sql --no-index --find-links ../target/wheels --force-reinstall", + "style": { + "code": true + } + }, + { + "type": "text", + "text": "\nIt is installed and thus skipped/passed when pip later checks if it needs to be installed." + } + ] + }, + { + "type": "rich_text_section", + "elements": [ + { + "type": "text", + "text": "That’s actually expected. You should build new wheel locally and then install it.\n\n" + } + ] + }, + { + "type": "rich_text_quote", + "elements": [ + { + "type": "text", + "text": "Now I have a second issue because it is expecting me to have mysqlclient-2.2.0 which seems to need a binary\n" + }, + { + "type": "text", + "text": "Command 'pkg-config --exists mysqlclient' returned non-zero exit status 127", + "style": { + "code": true + } + }, + { + "type": "text", + "text": "\nand\n" + }, + { + "type": "text", + "text": "Command 'pkg-config --exists mariadb' returned non-zero exit status 127", + "style": { + "code": true + } + }, + { + "type": "text", + "text": "\nI am on Ubuntu 22.04 in WSL2. Should I go to apt and grab me a mysql client?" + } + ] + }, + { + "type": "rich_text_section", + "elements": [ + { + "type": "text", + "text": "We’ve left some system specific configuration, e.g. mysqlclient, to users as it’s a bit aside from OpenLineage and more of general development task.\n\nprobably\n" + } + ] + }, + { + "type": "rich_text_preformatted", + "elements": [ + { + "type": "text", + "text": "sudo apt-get install python3-dev default-libmysqlclient-dev build-essential " + } + ], + "border": 0 + }, + { + "type": "rich_text_section", + "elements": [ + { + "type": "text", + "text": "should work" + } + ] + } + ] + } + ], + "team": "T01CWUYP5AR", + "thread_ts": "1695883240.832669", + "parent_user_id": "U05TZE47F2S" + }, + { + "client_msg_id": "2dac18f3-26e2-404e-97ba-7a7cf81b0803", + "type": "message", + "text": "I just realized that I should probably skip setting up my wsl and just run the tests in the docker setup you prepared", + "user": "U05TZE47F2S", + "ts": "1695886324.782519", + "blocks": [ + { + "type": "rich_text", + "block_id": "x/lLc", + "elements": [ + { + "type": "rich_text_section", + "elements": [ + { + "type": "text", + "text": "I just realized that I should probably skip setting up my wsl and just run the tests in the docker setup you prepared" + } + ] + } + ] + } + ], + "team": "T01CWUYP5AR", + "thread_ts": "1695883240.832669", + "parent_user_id": "U05TZE47F2S" + }, + { + "client_msg_id": "08cfc36c-c3da-4f09-9d87-6da0ea9adc64", + "type": "message", + "text": "You could do that as well but if you want to test your changes vs many Airflow versions that wouldn’t be possible I think (run them with tox btw)", + "user": "U02S6F54MAB", + "ts": "1695886546.366809", + "blocks": [ + { + "type": "rich_text", + "block_id": "hmzbn", + "elements": [ + { + "type": "rich_text_section", + "elements": [ + { + "type": "text", + "text": "You could do that as well but if you want to test your changes vs many Airflow versions that wouldn’t be possible I think (run them with tox btw)" + } + ] + } + ] + } + ], + "team": "T01CWUYP5AR", + "thread_ts": "1695883240.832669", + "parent_user_id": "U05TZE47F2S" + }, + { + "client_msg_id": "e1c50bc4-a763-4158-96ef-72e326321ef9", + "type": "message", + "text": "This is starting to feel like a rabbit hole :disappointed:\n\nWhen I run tox, I get a lot of build errors\n• client needs to be built\n• sql needs to be built to a different target than its readme says\n• a lot of builds fail on cython_sources", + "user": "U05TZE47F2S", + "ts": "1695891279.619909", + "blocks": [ + { + "type": "rich_text", + "block_id": "CF/Le", + "elements": [ + { + "type": "rich_text_section", + "elements": [ + { + "type": "text", + "text": "This is starting to feel like a rabbit hole " + }, + { + "type": "emoji", + "name": "disappointed", + "unicode": "1f61e" + }, + { + "type": "text", + "text": "\n\nWhen I run tox, I get a lot of build errors\n" + } + ] + }, + { + "type": "rich_text_list", + "elements": [ + { + "type": "rich_text_section", + "elements": [ + { + "type": "text", + "text": "client needs to be built" + } + ] + }, + { + "type": "rich_text_section", + "elements": [ + { + "type": "text", + "text": "sql needs to be built to a different target than its readme says" + } + ] + }, + { + "type": "rich_text_section", + "elements": [ + { + "type": "text", + "text": "a lot of builds fail on cython_sources" + } + ] + } + ], + "style": "bullet", + "indent": 0, + "border": 0 + } + ] + } + ], + "team": "T01CWUYP5AR", + "edited": { + "user": "U05TZE47F2S", + "ts": "1695891407.000000" + }, + "thread_ts": "1695883240.832669", + "parent_user_id": "U05TZE47F2S" + }, + { + "client_msg_id": "8c61b68c-8503-43d7-88d9-ff71102bed16", + "type": "message", + "text": "would you like to share some exact log lines? I’ve never seen such errors, they probably are system specific", + "user": "U02S6F54MAB", + "ts": "1695892774.394699", + "blocks": [ + { + "type": "rich_text", + "block_id": "C6YGc", + "elements": [ + { + "type": "rich_text_section", + "elements": [ + { + "type": "text", + "text": "would you like to share some exact log lines? I’ve never seen such errors, they probably are system specific" + } + ] + } + ] + } + ], + "team": "T01CWUYP5AR", + "thread_ts": "1695883240.832669", + "parent_user_id": "U05TZE47F2S" + }, + { + "client_msg_id": "36f3f924-102b-4915-91dc-091cd44002d7", + "type": "message", + "text": "`Getting requirements to build wheel did not run successfully.`\n`│ exit code: 1`\n`╰─> [62 lines of output]`\n `/tmp/pip-build-env-q1pay0xo/overlay/lib/python3.10/site-packages/setuptools/config/setupcfg.py:293: _DeprecatedConfig: Deprecated config in `setup.cfg``\n `!!`\n \n `********************************************************************************`\n `The license_file parameter is deprecated, use license_files instead.`\n \n `By 2023-Oct-30, you need to update your project and remove deprecated calls`\n `or your builds will no longer be supported.`\n \n `See for details.`\n `********************************************************************************`\n \n `!!`\n `parsed = self.parsers.get(option_name, lambda x: x)(value)`\n `running egg_info`\n `writing lib3/PyYAML.egg-info/PKG-INFO`\n `writing dependency_links to lib3/PyYAML.egg-info/dependency_links.txt`\n `writing top-level names to lib3/PyYAML.egg-info/top_level.txt`\n `Traceback (most recent call last):`\n `File \"/home/obr_erikal/projects/OpenLineage/integration/airflow/.tox/py3-airflow-2.1.4/lib/python3.10/site-packages/pip/_vendor/pyproject_hooks/_in_process/_in_process.py\", line 353, in <module>`\n `main()`\n `File \"/home/obr_erikal/projects/OpenLineage/integration/airflow/.tox/py3-airflow-2.1.4/lib/python3.10/site-packages/pip/_vendor/pyproject_hooks/_in_process/_in_process.py\", line 335, in main`\n `json_out['return_val'] = hook(**hook_input['kwargs'])`\n `File \"/home/obr_erikal/projects/OpenLineage/integration/airflow/.tox/py3-airflow-2.1.4/lib/python3.10/site-packages/pip/_vendor/pyproject_hooks/_in_process/_in_process.py\", line 118, in get_requires_for_build_wheel`\n `return hook(config_settings)`\n `File \"/tmp/pip-build-env-q1pay0xo/overlay/lib/python3.10/site-packages/setuptools/build_meta.py\", line 355, in get_requires_for_build_wheel`\n `return self._get_build_requires(config_settings, requirements=['wheel'])`\n `File \"/tmp/pip-build-env-q1pay0xo/overlay/lib/python3.10/site-packages/setuptools/build_meta.py\", line 325, in _get_build_requires`\n `self.run_setup()`\n `File \"/tmp/pip-build-env-q1pay0xo/overlay/lib/python3.10/site-packages/setuptools/build_meta.py\", line 341, in run_setup`\n `exec(code, locals())`\n `File \"<string>\", line 271, in <module>`\n `File \"/tmp/pip-build-env-q1pay0xo/overlay/lib/python3.10/site-packages/setuptools/__init__.py\", line 103, in setup`\n `return distutils.core.setup(**attrs)`\n `File \"/tmp/pip-build-env-q1pay0xo/overlay/lib/python3.10/site-packages/setuptools/_distutils/core.py\", line 185, in setup`\n `return run_commands(dist)`\n `File \"/tmp/pip-build-env-q1pay0xo/overlay/lib/python3.10/site-packages/setuptools/_distutils/core.py\", line 201, in run_commands`\n `dist.run_commands()`\n `File \"/tmp/pip-build-env-q1pay0xo/overlay/lib/python3.10/site-packages/setuptools/_distutils/dist.py\", line 969, in run_commands`\n `self.run_command(cmd)`\n `File \"/tmp/pip-build-env-q1pay0xo/overlay/lib/python3.10/site-packages/setuptools/dist.py\", line 989, in run_command`\n `super().run_command(command)`\n `File \"/tmp/pip-build-env-q1pay0xo/overlay/lib/python3.10/site-packages/setuptools/_distutils/dist.py\", line 988, in run_command`\n `cmd_obj.run()`\n `File \"/tmp/pip-build-env-q1pay0xo/overlay/lib/python3.10/site-packages/setuptools/command/egg_info.py\", line 318, in run`\n `self.find_sources()`\n `File \"/tmp/pip-build-env-q1pay0xo/overlay/lib/python3.10/site-packages/setuptools/command/egg_info.py\", line 326, in find_sources`\n `mm.run()`\n `File \"/tmp/pip-build-env-q1pay0xo/overlay/lib/python3.10/site-packages/setuptools/command/egg_info.py\", line 548, in run`\n `self.add_defaults()`\n `File \"/tmp/pip-build-env-q1pay0xo/overlay/lib/python3.10/site-packages/setuptools/command/egg_info.py\", line 586, in add_defaults`\n `sdist.add_defaults(self)`\n `File \"/tmp/pip-build-env-q1pay0xo/overlay/lib/python3.10/site-packages/setuptools/command/sdist.py\", line 113, in add_defaults`\n `super().add_defaults()`\n `File \"/tmp/pip-build-env-q1pay0xo/overlay/lib/python3.10/site-packages/setuptools/_distutils/command/sdist.py\", line 251, in add_defaults`\n `self._add_defaults_ext()`\n `File \"/tmp/pip-build-env-q1pay0xo/overlay/lib/python3.10/site-packages/setuptools/_distutils/command/sdist.py\", line 336, in _add_defaults_ext`\n `self.filelist.extend(build_ext.get_source_files())`\n `File \"<string>\", line 201, in get_source_files`\n `File \"/tmp/pip-build-env-q1pay0xo/overlay/lib/python3.10/site-packages/setuptools/_distutils/cmd.py\", line 107, in __getattr__`\n `raise AttributeError(attr)`\n `AttributeError: cython_sources`\n `[end of output]`\n\n`note: This error originates from a subprocess, and is likely not a problem with pip.`\n`py3-airflow-2.1.4: exit 1 (7.85 seconds) /home/obr_erikal/projects/OpenLineage/integration/airflow> python -m pip install --find-links target/wheels/ --find-links ../sql/iface-py/target/wheels --use-deprecated=legacy-resolver --constraint= apache-airflow==2.1.4 'mypy>=0.9.6' pytest pytest-mock -r dev-requirements.txt pid=368621`\n`py3-airflow-2.1.4: FAIL ✖ in 7.92 seconds`", + "user": "U05TZE47F2S", + "ts": "1695897948.265179", + "blocks": [ + { + "type": "rich_text", + "block_id": "V/odg", + "elements": [ + { + "type": "rich_text_section", + "elements": [ + { + "type": "text", + "text": "Getting requirements to build wheel did not run successfully.", + "style": { + "code": true + } + }, + { + "type": "text", + "text": "\n" + }, + { + "type": "text", + "text": "│ exit code: 1", + "style": { + "code": true + } + }, + { + "type": "text", + "text": "\n" + }, + { + "type": "text", + "text": "╰─> [62 lines of output]", + "style": { + "code": true + } + }, + { + "type": "text", + "text": "\n" + }, + { + "type": "text", + "text": " /tmp/pip-build-env-q1pay0xo/overlay/lib/python3.10/site-packages/setuptools/config/setupcfg.py:293: _DeprecatedConfig: Deprecated config in `setup.cfg`", + "style": { + "code": true + } + }, + { + "type": "text", + "text": "\n" + }, + { + "type": "text", + "text": " !!", + "style": { + "code": true + } + }, + { + "type": "text", + "text": "\n" + }, + { + "type": "text", + "text": " ", + "style": { + "code": true + } + }, + { + "type": "text", + "text": "\n" + }, + { + "type": "text", + "text": " ********************************************************************************", + "style": { + "code": true + } + }, + { + "type": "text", + "text": "\n" + }, + { + "type": "text", + "text": " The license_file parameter is deprecated, use license_files instead.", + "style": { + "code": true + } + }, + { + "type": "text", + "text": "\n" + }, + { + "type": "text", + "text": " ", + "style": { + "code": true + } + }, + { + "type": "text", + "text": "\n" + }, + { + "type": "text", + "text": " By 2023-Oct-30, you need to update your project and remove deprecated calls", + "style": { + "code": true + } + }, + { + "type": "text", + "text": "\n" + }, + { + "type": "text", + "text": " or your builds will no longer be supported.", + "style": { + "code": true + } + }, + { + "type": "text", + "text": "\n" + }, + { + "type": "text", + "text": " ", + "style": { + "code": true + } + }, + { + "type": "text", + "text": "\n" + }, + { + "type": "text", + "text": " See ", + "style": { + "code": true + } + }, + { + "type": "link", + "url": "https://setuptools.pypa.io/en/latest/userguide/declarative_config.html", + "style": { + "code": true + } + }, + { + "type": "text", + "text": " for details.", + "style": { + "code": true + } + }, + { + "type": "text", + "text": "\n" + }, + { + "type": "text", + "text": " ********************************************************************************", + "style": { + "code": true + } + }, + { + "type": "text", + "text": "\n" + }, + { + "type": "text", + "text": " ", + "style": { + "code": true + } + }, + { + "type": "text", + "text": "\n" + }, + { + "type": "text", + "text": " !!", + "style": { + "code": true + } + }, + { + "type": "text", + "text": "\n" + }, + { + "type": "text", + "text": " parsed = self.parsers.get(option_name, lambda x: x)(value)", + "style": { + "code": true + } + }, + { + "type": "text", + "text": "\n" + }, + { + "type": "text", + "text": " running egg_info", + "style": { + "code": true + } + }, + { + "type": "text", + "text": "\n" + }, + { + "type": "text", + "text": " writing lib3/PyYAML.egg-info/PKG-INFO", + "style": { + "code": true + } + }, + { + "type": "text", + "text": "\n" + }, + { + "type": "text", + "text": " writing dependency_links to lib3/PyYAML.egg-info/dependency_links.txt", + "style": { + "code": true + } + }, + { + "type": "text", + "text": "\n" + }, + { + "type": "text", + "text": " writing top-level names to lib3/PyYAML.egg-info/top_level.txt", + "style": { + "code": true + } + }, + { + "type": "text", + "text": "\n" + }, + { + "type": "text", + "text": " Traceback (most recent call last):", + "style": { + "code": true + } + }, + { + "type": "text", + "text": "\n" + }, + { + "type": "text", + "text": " File \"/home/obr_erikal/projects/OpenLineage/integration/airflow/.tox/py3-airflow-2.1.4/lib/python3.10/site-packages/pip/_vendor/pyproject_hooks/_in_process/_in_process.py\", line 353, in ", + "style": { + "code": true + } + }, + { + "type": "text", + "text": "\n" + }, + { + "type": "text", + "text": " main()", + "style": { + "code": true + } + }, + { + "type": "text", + "text": "\n" + }, + { + "type": "text", + "text": " File \"/home/obr_erikal/projects/OpenLineage/integration/airflow/.tox/py3-airflow-2.1.4/lib/python3.10/site-packages/pip/_vendor/pyproject_hooks/_in_process/_in_process.py\", line 335, in main", + "style": { + "code": true + } + }, + { + "type": "text", + "text": "\n" + }, + { + "type": "text", + "text": " json_out['return_val'] = hook(**hook_input['kwargs'])", + "style": { + "code": true + } + }, + { + "type": "text", + "text": "\n" + }, + { + "type": "text", + "text": " File \"/home/obr_erikal/projects/OpenLineage/integration/airflow/.tox/py3-airflow-2.1.4/lib/python3.10/site-packages/pip/_vendor/pyproject_hooks/_in_process/_in_process.py\", line 118, in get_requires_for_build_wheel", + "style": { + "code": true + } + }, + { + "type": "text", + "text": "\n" + }, + { + "type": "text", + "text": " return hook(config_settings)", + "style": { + "code": true + } + }, + { + "type": "text", + "text": "\n" + }, + { + "type": "text", + "text": " File \"/tmp/pip-build-env-q1pay0xo/overlay/lib/python3.10/site-packages/setuptools/build_meta.py\", line 355, in get_requires_for_build_wheel", + "style": { + "code": true + } + }, + { + "type": "text", + "text": "\n" + }, + { + "type": "text", + "text": " return self._get_build_requires(config_settings, requirements=['wheel'])", + "style": { + "code": true + } + }, + { + "type": "text", + "text": "\n" + }, + { + "type": "text", + "text": " File \"/tmp/pip-build-env-q1pay0xo/overlay/lib/python3.10/site-packages/setuptools/build_meta.py\", line 325, in _get_build_requires", + "style": { + "code": true + } + }, + { + "type": "text", + "text": "\n" + }, + { + "type": "text", + "text": " self.run_setup()", + "style": { + "code": true + } + }, + { + "type": "text", + "text": "\n" + }, + { + "type": "text", + "text": " File \"/tmp/pip-build-env-q1pay0xo/overlay/lib/python3.10/site-packages/setuptools/build_meta.py\", line 341, in run_setup", + "style": { + "code": true + } + }, + { + "type": "text", + "text": "\n" + }, + { + "type": "text", + "text": " exec(code, locals())", + "style": { + "code": true + } + }, + { + "type": "text", + "text": "\n" + }, + { + "type": "text", + "text": " File \"\", line 271, in ", + "style": { + "code": true + } + }, + { + "type": "text", + "text": "\n" + }, + { + "type": "text", + "text": " File \"/tmp/pip-build-env-q1pay0xo/overlay/lib/python3.10/site-packages/setuptools/__init__.py\", line 103, in setup", + "style": { + "code": true + } + }, + { + "type": "text", + "text": "\n" + }, + { + "type": "text", + "text": " return distutils.core.setup(**attrs)", + "style": { + "code": true + } + }, + { + "type": "text", + "text": "\n" + }, + { + "type": "text", + "text": " File \"/tmp/pip-build-env-q1pay0xo/overlay/lib/python3.10/site-packages/setuptools/_distutils/core.py\", line 185, in setup", + "style": { + "code": true + } + }, + { + "type": "text", + "text": "\n" + }, + { + "type": "text", + "text": " return run_commands(dist)", + "style": { + "code": true + } + }, + { + "type": "text", + "text": "\n" + }, + { + "type": "text", + "text": " File \"/tmp/pip-build-env-q1pay0xo/overlay/lib/python3.10/site-packages/setuptools/_distutils/core.py\", line 201, in run_commands", + "style": { + "code": true + } + }, + { + "type": "text", + "text": "\n" + }, + { + "type": "text", + "text": " dist.run_commands()", + "style": { + "code": true + } + }, + { + "type": "text", + "text": "\n" + }, + { + "type": "text", + "text": " File \"/tmp/pip-build-env-q1pay0xo/overlay/lib/python3.10/site-packages/setuptools/_distutils/dist.py\", line 969, in run_commands", + "style": { + "code": true + } + }, + { + "type": "text", + "text": "\n" + }, + { + "type": "text", + "text": " self.run_command(cmd)", + "style": { + "code": true + } + }, + { + "type": "text", + "text": "\n" + }, + { + "type": "text", + "text": " File \"/tmp/pip-build-env-q1pay0xo/overlay/lib/python3.10/site-packages/setuptools/dist.py\", line 989, in run_command", + "style": { + "code": true + } + }, + { + "type": "text", + "text": "\n" + }, + { + "type": "text", + "text": " super().run_command(command)", + "style": { + "code": true + } + }, + { + "type": "text", + "text": "\n" + }, + { + "type": "text", + "text": " File \"/tmp/pip-build-env-q1pay0xo/overlay/lib/python3.10/site-packages/setuptools/_distutils/dist.py\", line 988, in run_command", + "style": { + "code": true + } + }, + { + "type": "text", + "text": "\n" + }, + { + "type": "text", + "text": " cmd_obj.run()", + "style": { + "code": true + } + }, + { + "type": "text", + "text": "\n" + }, + { + "type": "text", + "text": " File \"/tmp/pip-build-env-q1pay0xo/overlay/lib/python3.10/site-packages/setuptools/command/egg_info.py\", line 318, in run", + "style": { + "code": true + } + }, + { + "type": "text", + "text": "\n" + }, + { + "type": "text", + "text": " self.find_sources()", + "style": { + "code": true + } + }, + { + "type": "text", + "text": "\n" + }, + { + "type": "text", + "text": " File \"/tmp/pip-build-env-q1pay0xo/overlay/lib/python3.10/site-packages/setuptools/command/egg_info.py\", line 326, in find_sources", + "style": { + "code": true + } + }, + { + "type": "text", + "text": "\n" + }, + { + "type": "text", + "text": " mm.run()", + "style": { + "code": true + } + }, + { + "type": "text", + "text": "\n" + }, + { + "type": "text", + "text": " File \"/tmp/pip-build-env-q1pay0xo/overlay/lib/python3.10/site-packages/setuptools/command/egg_info.py\", line 548, in run", + "style": { + "code": true + } + }, + { + "type": "text", + "text": "\n" + }, + { + "type": "text", + "text": " self.add_defaults()", + "style": { + "code": true + } + }, + { + "type": "text", + "text": "\n" + }, + { + "type": "text", + "text": " File \"/tmp/pip-build-env-q1pay0xo/overlay/lib/python3.10/site-packages/setuptools/command/egg_info.py\", line 586, in add_defaults", + "style": { + "code": true + } + }, + { + "type": "text", + "text": "\n" + }, + { + "type": "text", + "text": " sdist.add_defaults(self)", + "style": { + "code": true + } + }, + { + "type": "text", + "text": "\n" + }, + { + "type": "text", + "text": " File \"/tmp/pip-build-env-q1pay0xo/overlay/lib/python3.10/site-packages/setuptools/command/sdist.py\", line 113, in add_defaults", + "style": { + "code": true + } + }, + { + "type": "text", + "text": "\n" + }, + { + "type": "text", + "text": " super().add_defaults()", + "style": { + "code": true + } + }, + { + "type": "text", + "text": "\n" + }, + { + "type": "text", + "text": " File \"/tmp/pip-build-env-q1pay0xo/overlay/lib/python3.10/site-packages/setuptools/_distutils/command/sdist.py\", line 251, in add_defaults", + "style": { + "code": true + } + }, + { + "type": "text", + "text": "\n" + }, + { + "type": "text", + "text": " self._add_defaults_ext()", + "style": { + "code": true + } + }, + { + "type": "text", + "text": "\n" + }, + { + "type": "text", + "text": " File \"/tmp/pip-build-env-q1pay0xo/overlay/lib/python3.10/site-packages/setuptools/_distutils/command/sdist.py\", line 336, in _add_defaults_ext", + "style": { + "code": true + } + }, + { + "type": "text", + "text": "\n" + }, + { + "type": "text", + "text": " self.filelist.extend(build_ext.get_source_files())", + "style": { + "code": true + } + }, + { + "type": "text", + "text": "\n" + }, + { + "type": "text", + "text": " File \"\", line 201, in get_source_files", + "style": { + "code": true + } + }, + { + "type": "text", + "text": "\n" + }, + { + "type": "text", + "text": " File \"/tmp/pip-build-env-q1pay0xo/overlay/lib/python3.10/site-packages/setuptools/_distutils/cmd.py\", line 107, in __getattr__", + "style": { + "code": true + } + }, + { + "type": "text", + "text": "\n" + }, + { + "type": "text", + "text": " raise AttributeError(attr)", + "style": { + "code": true + } + }, + { + "type": "text", + "text": "\n" + }, + { + "type": "text", + "text": " AttributeError: cython_sources", + "style": { + "code": true + } + }, + { + "type": "text", + "text": "\n" + }, + { + "type": "text", + "text": " [end of output]", + "style": { + "code": true + } + }, + { + "type": "text", + "text": "\n\n" + }, + { + "type": "text", + "text": "note: This error originates from a subprocess, and is likely not a problem with pip.", + "style": { + "code": true + } + }, + { + "type": "text", + "text": "\n" + }, + { + "type": "text", + "text": "py3-airflow-2.1.4: exit 1 (7.85 seconds) /home/obr_erikal/projects/OpenLineage/integration/airflow> python -m pip install --find-links target/wheels/ --find-links ../sql/iface-py/target/wheels --use-deprecated=legacy-resolver --constraint=", + "style": { + "code": true + } + }, + { + "type": "link", + "url": "https://raw.githubusercontent.com/apache/airflow/constraints-2.1.4/constraints-3.8.txt", + "style": { + "code": true + } + }, + { + "type": "text", + "text": " apache-airflow==2.1.4 'mypy>=0.9.6' pytest pytest-mock -r dev-requirements.txt pid=368621", + "style": { + "code": true + } + }, + { + "type": "text", + "text": "\n" + }, + { + "type": "text", + "text": "py3-airflow-2.1.4: FAIL ✖ in 7.92 seconds", + "style": { + "code": true + } + } + ] + } + ] + } + ], + "team": "T01CWUYP5AR", + "thread_ts": "1695883240.832669", + "parent_user_id": "U05TZE47F2S" + }, + { + "client_msg_id": "c8a1bf48-bf7d-43d9-b0a0-13992b43c55b", + "type": "message", + "text": "Then, for the actual error in my PR: Evidently you are not using isort, so what linter/fixer should I use for imports?", + "user": "U05TZE47F2S", + "ts": "1695898434.713869", + "blocks": [ + { + "type": "rich_text", + "block_id": "0aovN", + "elements": [ + { + "type": "rich_text_section", + "elements": [ + { + "type": "text", + "text": "Then, for the actual error in my PR: Evidently you are not using isort, so what linter/fixer should I use for imports?" + } + ] + } + ] + } + ], + "team": "T01CWUYP5AR", + "thread_ts": "1695883240.832669", + "parent_user_id": "U05TZE47F2S" + }, + { + "client_msg_id": "fdedfa26-6342-4bd3-9f99-f4c9d76cf8ab", + "type": "message", + "text": "for the error - I think there’s a mistake in the docs. Could you please run `maturin build --out target/wheels` as a temp solution?", + "user": "U02S6F54MAB", + "ts": "1695898695.351679", + "blocks": [ + { + "type": "rich_text", + "block_id": "Lh5PY", + "elements": [ + { + "type": "rich_text_section", + "elements": [ + { + "type": "text", + "text": "for the error - I think there’s a mistake in the docs. Could you please run " + }, + { + "type": "text", + "text": "maturin build --out target/wheels", + "style": { + "code": true + } + }, + { + "type": "text", + "text": " as a temp solution?" + } + ] + } + ] + } + ], + "team": "T01CWUYP5AR", + "thread_ts": "1695883240.832669", + "parent_user_id": "U05TZE47F2S", + "reactions": [ + { + "name": "eyes", + "users": [ + "U05TZE47F2S" + ], + "count": 1 + } + ] + }, + { + "client_msg_id": "222d9ab1-faeb-4336-9f2c-e56f797fa7fd", + "type": "message", + "text": "we’re using `ruff` , tox runs it as one of commands", + "user": "U02S6F54MAB", + "ts": "1695898737.117519", + "blocks": [ + { + "type": "rich_text", + "block_id": "Kpa7W", + "elements": [ + { + "type": "rich_text_section", + "elements": [ + { + "type": "text", + "text": "we’re using " + }, + { + "type": "text", + "text": "ruff", + "style": { + "code": true + } + }, + { + "type": "text", + "text": " , tox runs it as one of commands" + } + ] + } + ] + } + ], + "team": "T01CWUYP5AR", + "thread_ts": "1695883240.832669", + "parent_user_id": "U05TZE47F2S" + }, + { + "client_msg_id": "156461a2-52be-4b91-b7d0-47779d5dba7b", + "type": "message", + "text": "Not in the airflow folder?\n`OpenLineage/integration/airflow$ maturin build --out target/wheels` \n`:boom: maturin failed`\n `Caused by: pyproject.toml at /home/obr_erikal/projects/OpenLineage/integration/airflow/pyproject.toml is invalid`\n `Caused by: TOML parse error at line 1, column 1`\n `|`\n`1 | [tool.ruff]`\n `| ^`\n`missing field `build-system``", + "user": "U05TZE47F2S", + "ts": "1695898837.753129", + "blocks": [ + { + "type": "rich_text", + "block_id": "VOnd9", + "elements": [ + { + "type": "rich_text_section", + "elements": [ + { + "type": "text", + "text": "Not in the airflow folder?\n" + }, + { + "type": "text", + "text": "OpenLineage/integration/airflow$ maturin build --out target/wheels ", + "style": { + "code": true + } + }, + { + "type": "text", + "text": "\n" + }, + { + "type": "text", + "text": ":boom: maturin failed", + "style": { + "code": true + } + }, + { + "type": "text", + "text": "\n" + }, + { + "type": "text", + "text": " Caused by: pyproject.toml at /home/obr_erikal/projects/OpenLineage/integration/airflow/pyproject.toml is invalid", + "style": { + "code": true + } + }, + { + "type": "text", + "text": "\n" + }, + { + "type": "text", + "text": " Caused by: TOML parse error at line 1, column 1", + "style": { + "code": true + } + }, + { + "type": "text", + "text": "\n" + }, + { + "type": "text", + "text": " |", + "style": { + "code": true + } + }, + { + "type": "text", + "text": "\n" + }, + { + "type": "text", + "text": "1 | [tool.ruff]", + "style": { + "code": true + } + }, + { + "type": "text", + "text": "\n" + }, + { + "type": "text", + "text": " | ^", + "style": { + "code": true + } + }, + { + "type": "text", + "text": "\n" + }, + { + "type": "text", + "text": "missing field `build-system`", + "style": { + "code": true + } + } + ] + } + ] + } + ], + "team": "T01CWUYP5AR", + "thread_ts": "1695883240.832669", + "parent_user_id": "U05TZE47F2S" + }, + { + "client_msg_id": "1c728711-bb38-4a15-84a5-56bf67874db0", + "type": "message", + "text": "I meant change here \n\nso\n```cd iface-py\npython -m pip install maturin\nmaturin build --out ../target/wheels```\nbecomes\n```cd iface-py\npython -m pip install maturin\nmaturin build --out target/wheels```\ntox runs\n```install_command = python -m pip install {opts} --find-links target/wheels/ \\\n\t--find-links ../sql/iface-py/target/wheels```\nbut it should be\n```install_command = python -m pip install {opts} --find-links target/wheels/ \\\n\t--find-links ../sql/target/wheels```\nactually and I’m posting PR to fix that", + "user": "U02S6F54MAB", + "ts": "1695898952.354359", + "blocks": [ + { + "type": "rich_text", + "block_id": "uXCf0", + "elements": [ + { + "type": "rich_text_section", + "elements": [ + { + "type": "text", + "text": "I meant change here " + }, + { + "type": "link", + "url": "https://github.com/OpenLineage/OpenLineage/blob/main/integration/sql/README.md" + }, + { + "type": "text", + "text": "\n\nso\n" + } + ] + }, + { + "type": "rich_text_preformatted", + "elements": [ + { + "type": "text", + "text": "cd iface-py\npython -m pip install maturin\nmaturin build --out ../target/wheels" + } + ], + "border": 0 + }, + { + "type": "rich_text_section", + "elements": [ + { + "type": "text", + "text": "becomes\n" + } + ] + }, + { + "type": "rich_text_preformatted", + "elements": [ + { + "type": "text", + "text": "cd iface-py\npython -m pip install maturin\nmaturin build --out target/wheels" + } + ], + "border": 0 + }, + { + "type": "rich_text_section", + "elements": [ + { + "type": "text", + "text": "\ntox runs\n" + } + ] + }, + { + "type": "rich_text_preformatted", + "elements": [ + { + "type": "text", + "text": "install_command = python -m pip install {opts} --find-links target/wheels/ \\\n\t--find-links ../sql/iface-py/target/wheels" + } + ], + "border": 0 + }, + { + "type": "rich_text_section", + "elements": [ + { + "type": "text", + "text": "but it should be\n" + } + ] + }, + { + "type": "rich_text_preformatted", + "elements": [ + { + "type": "text", + "text": "install_command = python -m pip install {opts} --find-links target/wheels/ \\\n\t--find-links ../sql/target/wheels" + } + ], + "border": 0 + }, + { + "type": "rich_text_section", + "elements": [ + { + "type": "text", + "text": "actually and I’m posting PR to fix that" + } + ] + } + ] + } + ], + "team": "T01CWUYP5AR", + "thread_ts": "1695883240.832669", + "parent_user_id": "U05TZE47F2S" + }, + { + "client_msg_id": "46514f2a-7d79-4ac6-a150-067d469356cb", + "type": "message", + "text": "yes, that part I actually worked out myself, but the cython_sources error I fail to understand cause. I have python3-dev installed on WSL Ubuntu with python version 3.10.12 in a virtualenv. Anything in that that could cause issues?", + "user": "U05TZE47F2S", + "ts": "1695899112.541119", + "blocks": [ + { + "type": "rich_text", + "block_id": "QFkIE", + "elements": [ + { + "type": "rich_text_section", + "elements": [ + { + "type": "text", + "text": "yes, that part I actually worked out myself, but the cython_sources error I fail to understand cause. I have python3-dev installed on WSL Ubuntu with python version 3.10.12 in a virtualenv. Anything in that that could cause issues?" + } + ] + } + ] + } + ], + "team": "T01CWUYP5AR", + "edited": { + "user": "U05TZE47F2S", + "ts": "1695899144.000000" + }, + "thread_ts": "1695883240.832669", + "parent_user_id": "U05TZE47F2S" + }, + { + "client_msg_id": "6ed420b6-90de-429d-a937-0718804b69b7", + "type": "message", + "text": "looks like it has something to do with latest release of Cython?\n`pip install \"Cython<3\"` maybe solves the issue?", + "user": "U02S6F54MAB", + "ts": "1695899540.894739", + "blocks": [ + { + "type": "rich_text", + "block_id": "NoPO+", + "elements": [ + { + "type": "rich_text_section", + "elements": [ + { + "type": "text", + "text": "looks like it has something to do with latest release of Cython?\n" + }, + { + "type": "text", + "text": "pip install \"Cython<3\"", + "style": { + "code": true + } + }, + { + "type": "text", + "text": " maybe solves the issue?" + } + ] + } + ] + } + ], + "team": "T01CWUYP5AR", + "edited": { + "user": "U02S6F54MAB", + "ts": "1695899564.000000" + }, + "thread_ts": "1695883240.832669", + "parent_user_id": "U05TZE47F2S" + }, + { + "client_msg_id": "b2f96fa2-ffb1-4e1f-99ae-cb860c044e2e", + "type": "message", + "text": "I didnt have any cython before the install. Also no change. Could it be some update to setuptools itself? seems like the depreciation notice and the error is coming from inside setuptools", + "user": "U05TZE47F2S", + "ts": "1695899706.634219", + "blocks": [ + { + "type": "rich_text", + "block_id": "cJw/+", + "elements": [ + { + "type": "rich_text_section", + "elements": [ + { + "type": "text", + "text": "I didnt have any cython before the install. Also no change. Could it be some update to setuptools itself? seems like the depreciation notice and the error is coming from inside setuptools" + } + ] + } + ] + } + ], + "team": "T01CWUYP5AR", + "thread_ts": "1695883240.832669", + "parent_user_id": "U05TZE47F2S" + }, + { + "client_msg_id": "2d334a13-2319-40bf-b5cf-ef4d12311da3", + "type": "message", + "text": "(I.e. I tried the `pip install \"Cython<3\"` command without any change in the output )", + "user": "U05TZE47F2S", + "ts": "1695899819.474779", + "blocks": [ + { + "type": "rich_text", + "block_id": "KAczn", + "elements": [ + { + "type": "rich_text_section", + "elements": [ + { + "type": "text", + "text": "(I.e. I tried the " + }, + { + "type": "text", + "text": "pip install \"Cython<3\"", + "style": { + "code": true + } + }, + { + "type": "text", + "text": " command without any change in the output )" + } + ] + } + ] + } + ], + "team": "T01CWUYP5AR", + "thread_ts": "1695883240.832669", + "parent_user_id": "U05TZE47F2S" + }, + { + "client_msg_id": "b5f08bb3-a0f9-4590-b105-ae4aa5889047", + "type": "message", + "text": "Applying ruff lint on the converter.py file fixed the issue on the PR though so unless you have any feedback on the change itself, I will set it up on my own computer later instead (right now doing changes on behalf of a client on the clients computer)\n\nIf the issue persists on my own computer, I'll dig a bit further", + "user": "U05TZE47F2S", + "ts": "1695900030.125419", + "blocks": [ + { + "type": "rich_text", + "block_id": "mGhfl", + "elements": [ + { + "type": "rich_text_section", + "elements": [ + { + "type": "text", + "text": "Applying ruff lint on the converter.py file fixed the issue on the PR though so unless you have any feedback on the change itself, I will set it up on my own computer later instead (right now doing changes on behalf of a client on the clients computer)\n\nIf the issue persists on my own computer, I'll dig a bit further" + } + ] + } + ] + } + ], + "team": "T01CWUYP5AR", + "thread_ts": "1695883240.832669", + "parent_user_id": "U05TZE47F2S" + }, + { + "client_msg_id": "c292e86b-4c4e-42c2-a924-7a1704d48633", + "type": "message", + "text": "It’s a bit hard for me to find the root cause as I cannot reproduce this locally and CI works fine as well", + "user": "U02S6F54MAB", + "ts": "1695900063.737969", + "blocks": [ + { + "type": "rich_text", + "block_id": "khU2e", + "elements": [ + { + "type": "rich_text_section", + "elements": [ + { + "type": "text", + "text": "It’s a bit hard for me to find the root cause as I cannot reproduce this locally and CI works fine as well" + } + ] + } + ] + } + ], + "team": "T01CWUYP5AR", + "thread_ts": "1695883240.832669", + "parent_user_id": "U05TZE47F2S" + }, + { + "client_msg_id": "a7598c78-c096-4c38-aed9-b317179faa41", + "type": "message", + "text": "Yeah, I am thinking that if I run into the same problem \"at home\", I might find it worthwhile to understand the issue. Right now, the client only wants the fix.", + "user": "U05TZE47F2S", + "ts": "1695900161.809479", + "blocks": [ + { + "type": "rich_text", + "block_id": "gSQO4", + "elements": [ + { + "type": "rich_text_section", + "elements": [ + { + "type": "text", + "text": "Yeah, I am thinking that if I run into the same problem \"at home\", I might find it worthwhile to understand the issue. Right now, the client only wants the fix." + } + ] + } + ] + } + ], + "team": "T01CWUYP5AR", + "thread_ts": "1695883240.832669", + "parent_user_id": "U05TZE47F2S", + "reactions": [ + { + "name": "+1", + "users": [ + "U02S6F54MAB" + ], + "count": 1 + } + ] + }, + { + "client_msg_id": "fbc7a559-9f3c-416f-b059-92e372149d40", + "type": "message", + "text": "Is there an official release cycle?\n\nor more specific, given that the PRs are approved, how soon can they reach openlineage-dbt and apache-airflow-providers-openlineage ?", + "user": "U05TZE47F2S", + "ts": "1695900310.007789", + "blocks": [ + { + "type": "rich_text", + "block_id": "2bs2Z", + "elements": [ + { + "type": "rich_text_section", + "elements": [ + { + "type": "text", + "text": "Is there an official release cycle?\n\nor more specific, given that the PRs are approved, how soon can they reach openlineage-dbt and apache-airflow-providers-openlineage ?" + } + ] + } + ] + } + ], + "team": "T01CWUYP5AR", + "thread_ts": "1695883240.832669", + "parent_user_id": "U05TZE47F2S" + }, + { + "client_msg_id": "4b04171d-6c1e-4e68-841c-c9b081a882e2", + "type": "message", + "text": "we need to differentiate some things:\n1. OpenLineage repository:\n a. dbt integration - this is the only place where it is maintained\n b. Airflow integration - here we only keep backwards compatibility but generally speaking starting from Airflow 2.7+ we would like to do all the job in Airflow repo as OL Airflow provider\n2. Airflow repository - there’s only Airflow Openlineage provider compatible (and works best) with Airflow 2.7+\n\nwe have control over releases (obviously) in OL repo - it’s monthly cycle so beginning next week that should happen. There’s also a possibility to ask for ad-hoc release in <#C01CK9T7HKR|general> slack channel and with approvals of committers the new version is also released\n\n\nFor Airflow providers - the cycle is monthly as well", + "user": "U02S6F54MAB", + "ts": "1695900538.450809", + "blocks": [ + { + "type": "rich_text", + "block_id": "fiJyL", + "elements": [ + { + "type": "rich_text_section", + "elements": [ + { + "type": "text", + "text": "we need to differentiate some things:\n" + } + ] + }, + { + "type": "rich_text_list", + "elements": [ + { + "type": "rich_text_section", + "elements": [ + { + "type": "text", + "text": "OpenLineage repository:" + } + ] + } + ], + "style": "ordered", + "indent": 0, + "border": 0 + }, + { + "type": "rich_text_list", + "elements": [ + { + "type": "rich_text_section", + "elements": [ + { + "type": "text", + "text": "dbt integration - this is the only place where it is maintained" + } + ] + }, + { + "type": "rich_text_section", + "elements": [ + { + "type": "text", + "text": "Airflow integration - here we only keep backwards compatibility but generally speaking starting from Airflow 2.7+ we would like to do all the job in Airflow repo as OL Airflow provider" + } + ] + } + ], + "style": "ordered", + "indent": 1, + "border": 0 + }, + { + "type": "rich_text_section", + "elements": [ + { + "type": "text", + "text": "2. Airflow repository - there’s only Airflow Openlineage provider compatible (and works best) with Airflow 2.7+\n\nwe have control over releases (obviously) in OL repo - it’s monthly cycle so beginning next week that should happen. There’s also a possibility to ask for ad-hoc release in " + }, + { + "type": "channel", + "channel_id": "C01CK9T7HKR" + }, + { + "type": "text", + "text": " slack channel and with approvals of committers the new version is also released\n\n\nFor Airflow providers - the cycle is monthly as well" + } + ] + } + ] + } + ], + "team": "T01CWUYP5AR", + "thread_ts": "1695883240.832669", + "parent_user_id": "U05TZE47F2S" + }, + { + "client_msg_id": "9f0f472f-6d10-47b2-aa32-4b0a4958889f", + "type": "message", + "text": "it’s a bit complex for this split but needed temporarily", + "user": "U02S6F54MAB", + "ts": "1695900690.131759", + "blocks": [ + { + "type": "rich_text", + "block_id": "8QdUB", + "elements": [ + { + "type": "rich_text_section", + "elements": [ + { + "type": "text", + "text": "it’s a bit complex for this split but needed temporarily" + } + ] + } + ] + } + ], + "team": "T01CWUYP5AR", + "thread_ts": "1695883240.832669", + "parent_user_id": "U05TZE47F2S" + }, + { + "client_msg_id": "eefb4e06-c0ec-4dec-92d6-2a759cebc579", + "type": "message", + "text": "oh, I did the fix in the wrong place! The client is on airflow 2.7 and is using the provider. Is it syncing?", + "user": "U05TZE47F2S", + "ts": "1695900707.652359", + "blocks": [ + { + "type": "rich_text", + "block_id": "UxXth", + "elements": [ + { + "type": "rich_text_section", + "elements": [ + { + "type": "text", + "text": "oh, I did the fix in the wrong place! The client is on airflow 2.7 and is using the provider. Is it syncing?" + } + ] + } + ] + } + ], + "team": "T01CWUYP5AR", + "thread_ts": "1695883240.832669", + "parent_user_id": "U05TZE47F2S" + }, + { + "client_msg_id": "5eb99cad-605c-4e92-9bff-620ed0b7fd0c", + "type": "message", + "text": "it’s not, two separate places a~nd we haven’t even added the whole thing with converting _old_ lineage objects to OL specific~\n\nediting, that’s not true", + "user": "U02S6F54MAB", + "ts": "1695900748.805309", + "blocks": [ + { + "type": "rich_text", + "block_id": "dUcDZ", + "elements": [ + { + "type": "rich_text_section", + "elements": [ + { + "type": "text", + "text": "it’s not, two separate places a" + }, + { + "type": "text", + "text": "nd we haven’t even added the whole thing with converting ", + "style": { + "strike": true + } + }, + { + "type": "text", + "text": "old", + "style": { + "italic": true, + "strike": true + } + }, + { + "type": "text", + "text": " lineage objects to OL specific", + "style": { + "strike": true + } + }, + { + "type": "text", + "text": "\n\nediting, that’s not true" + } + ] + } + ] + } + ], + "team": "T01CWUYP5AR", + "edited": { + "user": "U02S6F54MAB", + "ts": "1695900809.000000" + }, + "thread_ts": "1695883240.832669", + "parent_user_id": "U05TZE47F2S" + }, + { + "client_msg_id": "337aedba-adc6-4a71-82f3-53d9d90dca20", + "type": "message", + "text": "the code’s here:\n", + "user": "U02S6F54MAB", + "ts": "1695900880.660149", + "blocks": [ + { + "type": "rich_text", + "block_id": "GosPQ", + "elements": [ + { + "type": "rich_text_section", + "elements": [ + { + "type": "text", + "text": "the code’s here:\n" + }, + { + "type": "link", + "url": "https://github.com/apache/airflow/blob/main/airflow/providers/openlineage/extractors/manager.py#L154" + } + ] + } + ] + } + ], + "team": "T01CWUYP5AR", + "attachments": [ + { + "id": 1, + "footer_icon": "https://slack.github.com/static/img/favicon-neutral.png", + "color": "24292f", + "bot_id": "B01VA0FB340", + "app_unfurl_url": "https://github.com/apache/airflow/blob/main/airflow/providers/openlineage/extractors/manager.py#L154", + "is_app_unfurl": true, + "app_id": "A01BP7R4KNY", + "fallback": "", + "text": "```\n def extract_inlets_and_outlets(\n```", + "title": "", + "footer": "", + "mrkdwn_in": [ + "text" + ] + } + ], + "thread_ts": "1695883240.832669", + "parent_user_id": "U05TZE47F2S" + }, + { + "client_msg_id": "7bd82ea4-7a58-424a-8a2a-eeb1077e7787", + "type": "message", + "text": "sorry I did not mention this earlier. we definitely need to add some guidance how to proceed with contributions to OL and Airflow OL provider", + "user": "U02S6F54MAB", + "ts": "1695900917.385629", + "blocks": [ + { + "type": "rich_text", + "block_id": "OcKXi", + "elements": [ + { + "type": "rich_text_section", + "elements": [ + { + "type": "text", + "text": "sorry I did not mention this earlier. we definitely need to add some guidance how to proceed with contributions to OL and Airflow OL provider" + } + ] + } + ] + } + ], + "team": "T01CWUYP5AR", + "edited": { + "user": "U02S6F54MAB", + "ts": "1695900922.000000" + }, + "thread_ts": "1695883240.832669", + "parent_user_id": "U05TZE47F2S" + }, + { + "client_msg_id": "f87b4734-d583-43ac-92ac-1346ae5907e2", + "type": "message", + "text": "anyway, the dbt fix is the blocking issue, so if that parts comes next week, there is no real urgency in getting the columns. It is a nice to have for our ingest parquet files.", + "user": "U05TZE47F2S", + "ts": "1695900970.017429", + "blocks": [ + { + "type": "rich_text", + "block_id": "6vGZz", + "elements": [ + { + "type": "rich_text_section", + "elements": [ + { + "type": "text", + "text": "anyway, the dbt fix is the blocking issue, so if that parts comes next week, there is no real urgency in getting the columns. It is a nice to have for our ingest parquet files." + } + ] + } + ] + } + ], + "team": "T01CWUYP5AR", + "thread_ts": "1695883240.832669", + "parent_user_id": "U05TZE47F2S" + }, + { + "client_msg_id": "20642f4a-bc49-4541-a38a-bbbc14a52989", + "type": "message", + "text": "may I ask if you use some custom operator / python operator there?", + "user": "U02S6F54MAB", + "ts": "1695901032.688969", + "blocks": [ + { + "type": "rich_text", + "block_id": "8RMyR", + "elements": [ + { + "type": "rich_text_section", + "elements": [ + { + "type": "text", + "text": "may I ask if you use some custom operator / python operator there?" + } + ] + } + ] + } + ], + "team": "T01CWUYP5AR", + "thread_ts": "1695883240.832669", + "parent_user_id": "U05TZE47F2S" + }, + { + "client_msg_id": "3645fbfb-9b0a-4657-bd09-83f45dde6d8f", + "type": "message", + "text": "yeah, taskflow with inlets/outlets", + "user": "U05TZE47F2S", + "ts": "1695901053.392859", + "blocks": [ + { + "type": "rich_text", + "block_id": "fX9wa", + "elements": [ + { + "type": "rich_text_section", + "elements": [ + { + "type": "text", + "text": "yeah, taskflow with inlets/outlets" + } + ] + } + ] + } + ], + "team": "T01CWUYP5AR", + "thread_ts": "1695883240.832669", + "parent_user_id": "U05TZE47F2S" + }, + { + "client_msg_id": "4cbccc34-41a5-4c99-9beb-332b8d5fbbd2", + "type": "message", + "text": "so we extract from sources and use pyarrow to create parquet files in storage that an mssql-server can use as external tables", + "user": "U05TZE47F2S", + "ts": "1695901118.372199", + "blocks": [ + { + "type": "rich_text", + "block_id": "e/gWK", + "elements": [ + { + "type": "rich_text_section", + "elements": [ + { + "type": "text", + "text": "so we extract from sources and use pyarrow to create parquet files in storage that an mssql-server can use as external tables" + } + ] + } + ] + } + ], + "team": "T01CWUYP5AR", + "thread_ts": "1695883240.832669", + "parent_user_id": "U05TZE47F2S" + }, + { + "client_msg_id": "37037100-d69f-4187-8647-bcc8f408c18e", + "type": "message", + "text": "awesome :+1:\nwe have plans to integrate more with Python operator as well but not earlier than in Airflow 2.8", + "user": "U02S6F54MAB", + "ts": "1695901194.871609", + "blocks": [ + { + "type": "rich_text", + "block_id": "jCopB", + "elements": [ + { + "type": "rich_text_section", + "elements": [ + { + "type": "text", + "text": "awesome " + }, + { + "type": "emoji", + "name": "+1", + "unicode": "1f44d" + }, + { + "type": "text", + "text": "\nwe have plans to integrate more with Python operator as well but not earlier than in Airflow 2.8" + } + ] + } + ] + } + ], + "team": "T01CWUYP5AR", + "thread_ts": "1695883240.832669", + "parent_user_id": "U05TZE47F2S" + }, + { + "client_msg_id": "f9d22aa8-1c13-4c08-9460-e37a7993f0d9", + "type": "message", + "text": "I guess writing a generic extractor for the python operator is quite hard, but if you could support some inlet/outlet type for tabular fileformat / their python libraries like pyarrow or maybe even pandas and document it, I think a lot of people would understand how to use them", + "user": "U05TZE47F2S", + "ts": "1695901421.859499", + "blocks": [ + { + "type": "rich_text", + "block_id": "0a6uJ", + "elements": [ + { + "type": "rich_text_section", + "elements": [ + { + "type": "text", + "text": "I guess writing a generic extractor for the python operator is quite hard, but if you could support some inlet/outlet type for tabular fileformat / their python libraries like pyarrow or maybe even pandas and document it, I think a lot of people would understand how to use them" + } + ] + } + ] + } + ], + "team": "T01CWUYP5AR", + "thread_ts": "1695883240.832669", + "parent_user_id": "U05TZE47F2S", + "reactions": [ + { + "name": "heavy_plus_sign", + "users": [ + "U01HNKK4XAM" + ], + "count": 1 + } + ] + } + ] + }, + { + "client_msg_id": "cdfcae6a-f1f7-4075-826b-4f526d482042", + "type": "message", + "text": "Hi folks, am I correct in my observations that the Spark integration does not generate inputs and outputs for Kafka-to-Kafka pipelines?\n\n*EDIT:* Removed the crazy wall of text. Relevant GitHub issue is .", + "user": "U05FLJE4GDU", + "ts": "1695831785.042079", + "blocks": [ + { + "type": "rich_text", + "block_id": "3DLCm", + "elements": [ + { + "type": "rich_text_section", + "elements": [ + { + "type": "text", + "text": "Hi folks, am I correct in my observations that the Spark integration does not generate inputs and outputs for Kafka-to-Kafka pipelines?\n\n" + }, + { + "type": "text", + "text": "EDIT: ", + "style": { + "bold": true + } + }, + { + "type": "text", + "text": "Removed the crazy wall of text. Relevant GitHub issue is " + }, + { + "type": "link", + "url": "https://github.com/OpenLineage/OpenLineage/issues/2137", + "text": "here" + }, + { + "type": "text", + "text": "." + } + ] + } + ] + } + ], + "team": "T01CWUYP5AR", + "edited": { + "user": "U05FLJE4GDU", + "ts": "1695833102.000000" + }, + "attachments": [ + { + "id": 1, + "footer_icon": "https://slack.github.com/static/img/favicon-neutral.png", + "ts": 1695832992, + "color": "36a64f", + "bot_id": "B01VA0FB340", + "app_unfurl_url": "https://github.com/OpenLineage/OpenLineage/issues/2137", + "is_app_unfurl": true, + "app_id": "A01BP7R4KNY", + "fallback": "#2137 [SPARK] The integration fails to produce inputs and outputs in a Kafka-to-Kafka scenario", + "text": "*Problem Statement*\n\nAs the title suggests, when a Spark job is running in a streaming configuration that performs a Kafka-to-Kafka data flow, the integration fails to emit any events.\n\nHere's an example of an event that the connector emits:\n\nThe connector emits the following output:\n\n```\n23/09/27 18:17:22 DEBUG OpenLineageRunEventBuilder: Visiting query plan Optional[== Parsed Logical Plan ==\nWriteToMicroBatchDataSource org.apache.spark.sql.kafka010.KafkaSourceProvider$KafkaTable@2e679177, c3df858a-d481-456f-9665-3ff0c2d6d19a, [kafka.bootstrap.servers=localhost:9092,localhost:9093,localhost:9094, topic=target, checkpointLocation=/Users/dhawes/Projects/spark-streaming-openlineage], Append, 0\n+- StreamingDataSourceV2Relation [key#7, value#8, topic#9, partition#10, offset#11L, timestamp#12, timestampType#13], org.apache.spark.sql.kafka010.KafkaSourceProvider$KafkaScan@2095a606, KafkaV2[Subscribe[source]], {\"source\":{\"0\":11}}, {\"source\":{\"0\":11}}\n\n== Analyzed Logical Plan ==\nWriteToMicroBatchDataSource org.apache.spark.sql.kafka010.KafkaSourceProvider$KafkaTable@2e679177, c3df858a-d481-456f-9665-3ff0c2d6d19a, [kafka.bootstrap.servers=localhost:9092,localhost:9093,localhost:9094, topic=target, checkpointLocation=/Users/dhawes/Projects/spark-streaming-openlineage], Append, 0\n+- StreamingDataSourceV2Relation [key#7, value#8, topic#9, partition#10, offset#11L, timestamp#12, timestampType#13], org.apache.spark.sql.kafka010.KafkaSourceProvider$KafkaScan@2095a606, KafkaV2[Subscribe[source]], {\"source\":{\"0\":11}}, {\"source\":{\"0\":11}}\n\n== Optimized Logical Plan ==\nWriteToDataSourceV2 org.apache.spark.sql.execution.streaming.sources.MicroBatchWrite@6b69d31b\n+- StreamingDataSourceV2Relation [key#7, value#8, topic#9, partition#10, offset#11L, timestamp#12, timestampType#13], org.apache.spark.sql.kafka010.KafkaSourceProvider$KafkaScan@2095a606, KafkaV2[Subscribe[source]], {\"source\":{\"0\":11}}, {\"source\":{\"0\":11}}\n\n== Physical Plan ==\nWriteToDataSourceV2 org.apache.spark.sql.execution.streaming.sources.MicroBatchWrite@6b69d31b, org.apache.spark.sql.execution.datasources.v2.DataSourceV2Strategy$$Lambda$1945/0x000000c001c246c8@28b23a0a\n+- *(1) Project [key#7, value#8, topic#9, partition#10, offset#11L, timestamp#12, timestampType#13]\n +- MicroBatchScan[key#7, value#8, topic#9, partition#10, offset#11L, timestamp#12, timestampType#13] class org.apache.spark.sql.kafka010.KafkaSourceProvider$KafkaScan\n] with output dataset builders [, , , , , , , ]\n\n23/09/27 18:17:22 INFO ConsoleTransport: {\"eventTime\":\"2023-09-27T16:17:22.263Z\",\"producer\":\"\",\"schemaURL\":\"\",\"eventType\":\"COMPLETE\",\"run\":{\"runId\":\"3468112f-96c3-44b5-bd84-847c2d99db8f\",\"facets\":{\"spark_version\":{\"_producer\":\"\",\"_schemaURL\":\"\",\"spark-version\":\"3.3.0\",\"openlineage-spark-version\":\"1.2.2\"},\"processing_engine\":{\"_producer\":\"\",\"_schemaURL\":\"\",\"version\":\"3.3.0\",\"name\":\"spark\",\"openlineageAdapterVersion\":\"1.2.2\"},\"environment-properties\":{\"_producer\":\"\",\"_schemaURL\":\"\",\"environment-properties\":{}}}},\"job\":{\"namespace\":\"default\",\"name\":\"spark_streaming_example.write_to_data_source_v2\",\"facets\":{}},\"inputs\":[],\"outputs\":[]}\n```\n\nAs you can see, there are _zero_ inputs and outputs.\n\n*Stuff to reproduce*\n*Code*\n\n```\npackage streaming;\n\nimport org.apache.spark.sql.Dataset;\nimport org.apache.spark.sql.Row;\nimport org.apache.spark.sql.SparkSession;\nimport org.apache.spark.sql.streaming.StreamingQuery;\nimport org.apache.spark.sql.streaming.Trigger;\n\nimport java.util.concurrent.TimeUnit;\n\npublic class SparkStreamingExampleApplication {\n public static void main(String[] args) throws Exception {\n SparkSession spark = SparkSession\n .builder()\n .appName(\"spark-streaming-example\")\n .master(\"local\")\n .config(\"spark.ui.enabled\", false)\n .config(\"spark.jars.packages\", \"io.openlineage:openlineage-spark:1.2.2\")\n .config(\"spark.extraListeners\", \"io.openlineage.spark.agent.OpenLineageSparkListener\")\n .config(\"spark.openlineage.transport.type\", \"console\")\n .config(\"spark.openlineage.facets.disabled\", \"[spark_unknown;spark.logicalPlan]\")\n .getOrCreate();\n\n Dataset df = spark.readStream()\n .format(\"kafka\")\n .option(\"kafka.bootstrap.servers\", \"localhost:9092,localhost:9093,localhost:9094\")\n .option(\"subscribe\", \"source\")\n .load();\n\n StreamingQuery kafkaWriteQuery = df.writeStream()\n .outputMode(\"append\")\n .format(\"kafka\")\n .option(\"kafka.bootstrap.servers\", \"localhost:9092,localhost:9093,localhost:9094\")\n .option(\"topic\", \"target\")\n .option(\"checkpointLocation\", \"/Users/dhawes/Projects/spark-streaming-openlineage\")\n .trigger(Trigger.ProcessingTime(10, TimeUnit.SECONDS))\n .start();\n\n kafkaWriteQuery.awaitTermination();\n }\n}\n```\n\n*build.gradle.kts*\n\n```\nplugins {\n java\n application\n}\n\njava {\n sourceCompatibility = JavaVersion.VERSION_1_8\n targetCompatibility = JavaVersion.VERSION_1_8\n\n toolchain {\n languageVersion.set(JavaLanguageVersion.of(8))\n }\n}\n\nrepositories {\n mavenLocal()\n mavenCentral()\n}\n\ndependencies {\n implementation(\"org.apache.spark:spark-core_2.12:3.3.0\")\n implementation(\"org.apache.spark:spark-sql_2.12:3.3.0\")\n implementation(\"org.apache.spark:spark-sql-kafka-0-10_2.12:3.3.0\")\n implementation(\"org.apache.spark:spark-streaming_2.12:3.3.0\")\n implementation(\"org.apache.spark:spark-streaming-kafka-0-10_2.12:3.3.0\")\n implementation(\"io.openlineage:openlineage-spark:1.2.2\")\n}\n\napplication {\n mainClass = \"streaming.SparkStreamingExampleApplication\"\n}\n```\n\n*docker-compose.yaml*\n\n```\nversion: '3'\n\nservices:\n zookeeper:\n image: confluentinc/cp-zookeeper:latest\n container_name: 'spark-streaming-zookeeper'\n environment:\n ZOOKEEPER_CLIENT_PORT: 2181\n\n kafka1:\n image: 'confluentinc/cp-kafka:latest'\n container_name: 'spark-streaming-kafka-1'\n depends_on:\n - 'zookeeper'\n environment:\n KAFKA_BROKER_ID: 1\n KAFKA_ZOOKEEPER_CONNECT: zookeeper:2181\n KAFKA_ADVERTISED_LISTENERS: \n KAFKA_LISTENER_SECURITY_PROTOCOL_MAP: INSIDE:PLAINTEXT,OUTSIDE:PLAINTEXT\n KAFKA_INTER_BROKER_LISTENER_NAME: INSIDE\n KAFKA_LISTENERS: \n ports:\n - \"9092:9094\"\n\n kafka2:\n image: 'confluentinc/cp-kafka:latest'\n container_name: 'spark-streaming-kafka-2'\n depends_on:\n - 'zookeeper'\n environment:\n KAFKA_BROKER_ID: 2\n KAFKA_ZOOKEEPER_CONNECT: zookeeper:2181\n KAFKA_ADVERTISED_LISTENERS: \n KAFKA_LISTENER_SECURITY_PROTOCOL_MAP: INSIDE:PLAINTEXT,OUTSIDE:PLAINTEXT\n KAFKA_INTER_BROKER_LISTENER_NAME: INSIDE\n KAFKA_LISTENERS: \n ports:\n - \"9093:9095\"\n\n kafka3:\n image: 'confluentinc/cp-kafka:latest'\n container_name: 'spark-streaming-kafka-3'\n depends_on:\n - 'zookeeper'\n environment:\n KAFKA_BROKER_ID: 3\n KAFKA_ZOOKEEPER_CONNECT: zookeeper:2181\n KAFKA_ADVERTISED_LISTENERS: ", + "mrkdwn_in": [ + "text" + ] + } + ], + "thread_ts": "1695831785.042079", + "reply_count": 1, + "reply_users_count": 1, + "latest_reply": "1695883338.816999", + "reply_users": [ + "U02MK6YNAQ5" + ], + "is_locked": false, + "subscribed": false, + "reactions": [ + { + "name": "eyes", + "users": [ + "U02MK6YNAQ5" + ], + "count": 1 + } + ], + "replies": [ + { + "client_msg_id": "605d51fd-23f6-4f14-bd7b-8ec638b7ecac", + "type": "message", + "text": "responded within the issue", + "user": "U02MK6YNAQ5", + "ts": "1695883338.816999", + "blocks": [ + { + "type": "rich_text", + "block_id": "4EBnj", + "elements": [ + { + "type": "rich_text_section", + "elements": [ + { + "type": "text", + "text": "responded within the issue" + } + ] + } + ] + } + ], + "team": "T01CWUYP5AR", + "thread_ts": "1695831785.042079", + "parent_user_id": "U05FLJE4GDU" + } + ] + }, + { + "type": "message", + "text": "*Meetup recap: Toronto Meetup @ Airflow Summit, September 18, 2023*\nIt was great to see so many members of our community at this event! I counted 32 total attendees, with all but a handful being first-timers.\n*Topics included:*\n• Presentation on the history, architecture and roadmap of the project by <@U01DCLP0GU9> and <@U01HNKK4XAM> \n• Discussion of OpenLineage support in by <@U01DCMDFHBK> \n• Presentation by *Ye Liu* and *Ivan Perepelitca* from , the social platform for data, about their integration\n• Presentation by <@U02MK6YNAQ5> about the Spark integration\n• Presentation by <@U01RA9B5GG2> about the Apache Airflow Provider\nThanks to all the presenters and attendees with a shout out to <@U01HNKK4XAM> for the help with organizing and day-of logistics, <@U02S6F54MAB> for the help with set up/clean up, and <@U0323HG8C8H> for the crucial assist with the signup sheet.\nThis was our first meetup in Toronto, and we learned some valuable lessons about planning events in new cities — the first and foremost being to ask for a pic of the building! :slightly_smiling_face: But it seemed like folks were undeterred, and the space itself lived up to expectations.\nFor a recording and clips from the meetup, head over to our .\n*Upcoming events:*\n• October 5th in San Francisco: Marquez Meetup @ Astronomer (sign up )\n• November: Warsaw meetup (details, date TBA)\n• January: London meetup (details, date TBA)\nAre you interested in hosting or co-hosting an OpenLineage or Marquez meetup? DM me!", + "files": [ + { + "id": "F05TR87P1JB", + "created": 1695826325, + "timestamp": 1695826325, + "name": "IMG_5456.jpg", + "title": "IMG_5456.jpg", + "mimetype": "image/jpeg", + "filetype": "jpg", + "pretty_type": "JPEG", + "user": "U02LXF3HUN7", + "user_team": "T01CWUYP5AR", + "editable": false, + "size": 693622, + "mode": "hosted", + "is_external": false, + "external_type": "", + "is_public": true, + "public_url_shared": false, + "display_as_bot": false, + "username": "", + "url_private": "https://files.slack.com/files-pri/T01CWUYP5AR-F05TR87P1JB/img_5456.jpg", + "url_private_download": "https://files.slack.com/files-pri/T01CWUYP5AR-F05TR87P1JB/download/img_5456.jpg", + "media_display_type": "unknown", + "thumb_64": "https://files.slack.com/files-tmb/T01CWUYP5AR-F05TR87P1JB-007b1c010b/img_5456_64.jpg", + "thumb_80": "https://files.slack.com/files-tmb/T01CWUYP5AR-F05TR87P1JB-007b1c010b/img_5456_80.jpg", + "thumb_360": "https://files.slack.com/files-tmb/T01CWUYP5AR-F05TR87P1JB-007b1c010b/img_5456_360.jpg", + "thumb_360_w": 360, + "thumb_360_h": 270, + "thumb_480": "https://files.slack.com/files-tmb/T01CWUYP5AR-F05TR87P1JB-007b1c010b/img_5456_480.jpg", + "thumb_480_w": 480, + "thumb_480_h": 360, + "thumb_160": "https://files.slack.com/files-tmb/T01CWUYP5AR-F05TR87P1JB-007b1c010b/img_5456_160.jpg", + "thumb_720": "https://files.slack.com/files-tmb/T01CWUYP5AR-F05TR87P1JB-007b1c010b/img_5456_720.jpg", + "thumb_720_w": 720, + "thumb_720_h": 540, + "thumb_800": "https://files.slack.com/files-tmb/T01CWUYP5AR-F05TR87P1JB-007b1c010b/img_5456_800.jpg", + "thumb_800_w": 800, + "thumb_800_h": 600, + "thumb_960": "https://files.slack.com/files-tmb/T01CWUYP5AR-F05TR87P1JB-007b1c010b/img_5456_960.jpg", + "thumb_960_w": 960, + "thumb_960_h": 720, + "thumb_1024": "https://files.slack.com/files-tmb/T01CWUYP5AR-F05TR87P1JB-007b1c010b/img_5456_1024.jpg", + "thumb_1024_w": 1024, + "thumb_1024_h": 768, + "original_w": 4032, + "original_h": 3024, + "thumb_tiny": "AwAkADCTYzOdrEYFLiVeoDCnxDBf6gfpUgOehFJIGyESL/EpH4Zpw2t91gak69QDUUqqNpA70NApCnIppP8Ak0vJHHSjHY1BQgfDPx/Ead5g/u1VcyiQhASueM4pzebn5Dx74quYLFjev9002VgcfWq7faOzKKfHnb++JPpinzCsJFcrJcGMbgOgz609rhdzjj5emD1qCOJheebj93nrmoDEEcjJyD6UrAXyaAetIaXsakoYWNPC7gMk1G1Sp0FAEZ4ZvY/0oxlqG++/1/oKUffFAH//2Q==", + "permalink": "https://openlineage.slack.com/files/U02LXF3HUN7/F05TR87P1JB/img_5456.jpg", + "permalink_public": "https://slack-files.com/T01CWUYP5AR-F05TR87P1JB-7c7ec9ebe9", + "is_starred": false, + "has_rich_preview": false, + "file_access": "visible" + }, + { + "id": "F05TZ52T18W", + "created": 1695826339, + "timestamp": 1695826339, + "name": "IMG_5452.jpg", + "title": "IMG_5452.jpg", + "mimetype": "image/jpeg", + "filetype": "jpg", + "pretty_type": "JPEG", + "user": "U02LXF3HUN7", + "user_team": "T01CWUYP5AR", + "editable": false, + "size": 664390, + "mode": "hosted", + "is_external": false, + "external_type": "", + "is_public": true, + "public_url_shared": false, + "display_as_bot": false, + "username": "", + "url_private": "https://files.slack.com/files-pri/T01CWUYP5AR-F05TZ52T18W/img_5452.jpg", + "url_private_download": "https://files.slack.com/files-pri/T01CWUYP5AR-F05TZ52T18W/download/img_5452.jpg", + "media_display_type": "unknown", + "thumb_64": "https://files.slack.com/files-tmb/T01CWUYP5AR-F05TZ52T18W-717ba6ed78/img_5452_64.jpg", + "thumb_80": "https://files.slack.com/files-tmb/T01CWUYP5AR-F05TZ52T18W-717ba6ed78/img_5452_80.jpg", + "thumb_360": "https://files.slack.com/files-tmb/T01CWUYP5AR-F05TZ52T18W-717ba6ed78/img_5452_360.jpg", + "thumb_360_w": 360, + "thumb_360_h": 270, + "thumb_480": "https://files.slack.com/files-tmb/T01CWUYP5AR-F05TZ52T18W-717ba6ed78/img_5452_480.jpg", + "thumb_480_w": 480, + "thumb_480_h": 360, + "thumb_160": "https://files.slack.com/files-tmb/T01CWUYP5AR-F05TZ52T18W-717ba6ed78/img_5452_160.jpg", + "thumb_720": "https://files.slack.com/files-tmb/T01CWUYP5AR-F05TZ52T18W-717ba6ed78/img_5452_720.jpg", + "thumb_720_w": 720, + "thumb_720_h": 540, + "thumb_800": "https://files.slack.com/files-tmb/T01CWUYP5AR-F05TZ52T18W-717ba6ed78/img_5452_800.jpg", + "thumb_800_w": 800, + "thumb_800_h": 600, + "thumb_960": "https://files.slack.com/files-tmb/T01CWUYP5AR-F05TZ52T18W-717ba6ed78/img_5452_960.jpg", + "thumb_960_w": 960, + "thumb_960_h": 720, + "thumb_1024": "https://files.slack.com/files-tmb/T01CWUYP5AR-F05TZ52T18W-717ba6ed78/img_5452_1024.jpg", + "thumb_1024_w": 1024, + "thumb_1024_h": 768, + "original_w": 4032, + "original_h": 3024, + "thumb_tiny": "AwAkADByiUgkAEZ6Uu8Dh0ZfwzUkI/dJ7jNS7ciiw7lcBG+6wNGGHvUrQo3VRUTDY+0Mce9Jqw0wzSGlzntQVz61IyZSmANw496eGH94fnWak+84wBx3NHnNniOquKxpZXPUVDMAWXGO9UzNJniHj1qaM5AYkA46YpNgSk4PKt+AqKOcPMVyNuOARzmpQ6kZzVJFxccjvmkNE1L2pKXtSGITzTT1pT1pD1piHr06UxY135xg09elC/epDP/Z", + "permalink": "https://openlineage.slack.com/files/U02LXF3HUN7/F05TZ52T18W/img_5452.jpg", + "permalink_public": "https://slack-files.com/T01CWUYP5AR-F05TZ52T18W-581d9fefdf", + "is_starred": false, + "has_rich_preview": false, + "file_access": "visible" + }, + { + "id": "F05TR9U0V9V", + "created": 1695826788, + "timestamp": 1695826788, + "name": "20230918_171849.jpg", + "title": "20230918_171849.jpg", + "mimetype": "image/jpeg", + "filetype": "jpg", + "pretty_type": "JPEG", + "user": "U02LXF3HUN7", + "user_team": "T01CWUYP5AR", + "editable": false, + "size": 2000031, + "mode": "hosted", + "is_external": false, + "external_type": "", + "is_public": true, + "public_url_shared": false, + "display_as_bot": false, + "username": "", + "url_private": "https://files.slack.com/files-pri/T01CWUYP5AR-F05TR9U0V9V/20230918_171849.jpg", + "url_private_download": "https://files.slack.com/files-pri/T01CWUYP5AR-F05TR9U0V9V/download/20230918_171849.jpg", + "media_display_type": "unknown", + "thumb_64": "https://files.slack.com/files-tmb/T01CWUYP5AR-F05TR9U0V9V-cad8e2b752/20230918_171849_64.jpg", + "thumb_80": "https://files.slack.com/files-tmb/T01CWUYP5AR-F05TR9U0V9V-cad8e2b752/20230918_171849_80.jpg", + "thumb_360": "https://files.slack.com/files-tmb/T01CWUYP5AR-F05TR9U0V9V-cad8e2b752/20230918_171849_360.jpg", + "thumb_360_w": 360, + "thumb_360_h": 270, + "thumb_480": "https://files.slack.com/files-tmb/T01CWUYP5AR-F05TR9U0V9V-cad8e2b752/20230918_171849_480.jpg", + "thumb_480_w": 480, + "thumb_480_h": 360, + "thumb_160": "https://files.slack.com/files-tmb/T01CWUYP5AR-F05TR9U0V9V-cad8e2b752/20230918_171849_160.jpg", + "thumb_720": "https://files.slack.com/files-tmb/T01CWUYP5AR-F05TR9U0V9V-cad8e2b752/20230918_171849_720.jpg", + "thumb_720_w": 720, + "thumb_720_h": 540, + "thumb_800": "https://files.slack.com/files-tmb/T01CWUYP5AR-F05TR9U0V9V-cad8e2b752/20230918_171849_800.jpg", + "thumb_800_w": 800, + "thumb_800_h": 600, + "thumb_960": "https://files.slack.com/files-tmb/T01CWUYP5AR-F05TR9U0V9V-cad8e2b752/20230918_171849_960.jpg", + "thumb_960_w": 960, + "thumb_960_h": 720, + "thumb_1024": "https://files.slack.com/files-tmb/T01CWUYP5AR-F05TR9U0V9V-cad8e2b752/20230918_171849_1024.jpg", + "thumb_1024_w": 1024, + "thumb_1024_h": 768, + "original_w": 4032, + "original_h": 3024, + "thumb_tiny": "AwAkADCTIo69TSsuOaMD0FCVwbEwCOKMUoHJo4K5B9s0gECk9KRhjrinjkZFMlkC4B6+lAXGzOfLIUFT6kUwkZT5mO70pJQZItzElgM+nFRorTMQrFUFAyYuVbYDnB7c5FRO/wAmwAEA5yDUTEhsA00KS+1eSelMCzJJIcYO0AAYNLcKzSb+NuAKjMZB/etgKvY9aakjswGcKO1FwLAYhDim26hX4z6Uv/LM/Wli+9+NSMVoUyflFQKxRQy4BGRnFWm6mqh/1f4mmAN8xyTk04DBzTT2/Cn0gP/Z", + "permalink": "https://openlineage.slack.com/files/U02LXF3HUN7/F05TR9U0V9V/20230918_171849.jpg", + "permalink_public": "https://slack-files.com/T01CWUYP5AR-F05TR9U0V9V-f79f9ed473", + "is_starred": false, + "has_rich_preview": false, + "file_access": "visible" + }, + { + "id": "F05TZ6P1FD4", + "created": 1695826799, + "timestamp": 1695826799, + "name": "20230918_192522.jpg", + "title": "20230918_192522.jpg", + "mimetype": "image/jpeg", + "filetype": "jpg", + "pretty_type": "JPEG", + "user": "U02LXF3HUN7", + "user_team": "T01CWUYP5AR", + "editable": false, + "size": 1628494, + "mode": "hosted", + "is_external": false, + "external_type": "", + "is_public": true, + "public_url_shared": false, + "display_as_bot": false, + "username": "", + "url_private": "https://files.slack.com/files-pri/T01CWUYP5AR-F05TZ6P1FD4/20230918_192522.jpg", + "url_private_download": "https://files.slack.com/files-pri/T01CWUYP5AR-F05TZ6P1FD4/download/20230918_192522.jpg", + "media_display_type": "unknown", + "thumb_64": "https://files.slack.com/files-tmb/T01CWUYP5AR-F05TZ6P1FD4-ecc34f9575/20230918_192522_64.jpg", + "thumb_80": "https://files.slack.com/files-tmb/T01CWUYP5AR-F05TZ6P1FD4-ecc34f9575/20230918_192522_80.jpg", + "thumb_360": "https://files.slack.com/files-tmb/T01CWUYP5AR-F05TZ6P1FD4-ecc34f9575/20230918_192522_360.jpg", + "thumb_360_w": 360, + "thumb_360_h": 270, + "thumb_480": "https://files.slack.com/files-tmb/T01CWUYP5AR-F05TZ6P1FD4-ecc34f9575/20230918_192522_480.jpg", + "thumb_480_w": 480, + "thumb_480_h": 360, + "thumb_160": "https://files.slack.com/files-tmb/T01CWUYP5AR-F05TZ6P1FD4-ecc34f9575/20230918_192522_160.jpg", + "thumb_720": "https://files.slack.com/files-tmb/T01CWUYP5AR-F05TZ6P1FD4-ecc34f9575/20230918_192522_720.jpg", + "thumb_720_w": 720, + "thumb_720_h": 540, + "thumb_800": "https://files.slack.com/files-tmb/T01CWUYP5AR-F05TZ6P1FD4-ecc34f9575/20230918_192522_800.jpg", + "thumb_800_w": 800, + "thumb_800_h": 600, + "thumb_960": "https://files.slack.com/files-tmb/T01CWUYP5AR-F05TZ6P1FD4-ecc34f9575/20230918_192522_960.jpg", + "thumb_960_w": 960, + "thumb_960_h": 720, + "thumb_1024": "https://files.slack.com/files-tmb/T01CWUYP5AR-F05TZ6P1FD4-ecc34f9575/20230918_192522_1024.jpg", + "thumb_1024_w": 1024, + "thumb_1024_h": 768, + "original_w": 4032, + "original_h": 3024, + "thumb_tiny": "AwAkADCbb7UjDANTqBRIo2N9KAIxH7/pS+UPX9KcGBYrnnrSqwfOM8HHIpiGeUPX9KhBOKtVBj5BSYIlBwKJHHlsSeMVUvZMRBO7c/hRA4MSKSMjpQMFkZTuyc45pkcjI5Ysx+nemmOTY2D19T71EwaNgpbA4JxQxplv7SfST8qkDDyxk01PnAYcg96hut+wbcjntxSuBAvzJz26UIfmT/eFEf3TQn3k/wB4UwLBJyxzVe4OXBPpU/8AeqCb7w+lAE9uT5eOw6VHI7E8mpLf7hqF+tID/9k=", + "permalink": "https://openlineage.slack.com/files/U02LXF3HUN7/F05TZ6P1FD4/20230918_192522.jpg", + "permalink_public": "https://slack-files.com/T01CWUYP5AR-F05TZ6P1FD4-378e7de35f", + "is_starred": false, + "has_rich_preview": false, + "file_access": "visible" + }, + { + "id": "F05U5RQUTJ6", + "created": 1695826818, + "timestamp": 1695826818, + "name": "20230918_185612.jpg", + "title": "20230918_185612.jpg", + "mimetype": "image/jpeg", + "filetype": "jpg", + "pretty_type": "JPEG", + "user": "U02LXF3HUN7", + "user_team": "T01CWUYP5AR", + "editable": false, + "size": 1644913, + "mode": "hosted", + "is_external": false, + "external_type": "", + "is_public": true, + "public_url_shared": false, + "display_as_bot": false, + "username": "", + "url_private": "https://files.slack.com/files-pri/T01CWUYP5AR-F05U5RQUTJ6/20230918_185612.jpg", + "url_private_download": "https://files.slack.com/files-pri/T01CWUYP5AR-F05U5RQUTJ6/download/20230918_185612.jpg", + "media_display_type": "unknown", + "thumb_64": "https://files.slack.com/files-tmb/T01CWUYP5AR-F05U5RQUTJ6-af11e23ff9/20230918_185612_64.jpg", + "thumb_80": "https://files.slack.com/files-tmb/T01CWUYP5AR-F05U5RQUTJ6-af11e23ff9/20230918_185612_80.jpg", + "thumb_360": "https://files.slack.com/files-tmb/T01CWUYP5AR-F05U5RQUTJ6-af11e23ff9/20230918_185612_360.jpg", + "thumb_360_w": 360, + "thumb_360_h": 270, + "thumb_480": "https://files.slack.com/files-tmb/T01CWUYP5AR-F05U5RQUTJ6-af11e23ff9/20230918_185612_480.jpg", + "thumb_480_w": 480, + "thumb_480_h": 360, + "thumb_160": "https://files.slack.com/files-tmb/T01CWUYP5AR-F05U5RQUTJ6-af11e23ff9/20230918_185612_160.jpg", + "thumb_720": "https://files.slack.com/files-tmb/T01CWUYP5AR-F05U5RQUTJ6-af11e23ff9/20230918_185612_720.jpg", + "thumb_720_w": 720, + "thumb_720_h": 540, + "thumb_800": "https://files.slack.com/files-tmb/T01CWUYP5AR-F05U5RQUTJ6-af11e23ff9/20230918_185612_800.jpg", + "thumb_800_w": 800, + "thumb_800_h": 600, + "thumb_960": "https://files.slack.com/files-tmb/T01CWUYP5AR-F05U5RQUTJ6-af11e23ff9/20230918_185612_960.jpg", + "thumb_960_w": 960, + "thumb_960_h": 720, + "thumb_1024": "https://files.slack.com/files-tmb/T01CWUYP5AR-F05U5RQUTJ6-af11e23ff9/20230918_185612_1024.jpg", + "thumb_1024_w": 1024, + "thumb_1024_h": 768, + "original_w": 4032, + "original_h": 3024, + "thumb_tiny": "AwAkADB4BZ8Z7U/ym9adtxIvuDUgppCbIvLb2/KmupQZ4/KrGKinHyH6U2hJkWecYooVgzMB24NLioLHyNgqfc/yo83H8P5mqzRlnLFzk+lMlMceNyls+9NMVi2bjHUqPxqMzeZkKwPHQCmbI8AhV/Kop5PKK7ABnrxRcLE8TfOfSi4ZlQFCODzUYRgg2t83ck011kSInOT3AHai4EyjJqpcHMuD0FW0+9VSf/XGkhly3/1CnAziojiTllBP0qW3/wCPcfSol+7VIRJHy7fQUsjFc4psX32+gpZu/wCFJ7jR/9k=", + "permalink": "https://openlineage.slack.com/files/U02LXF3HUN7/F05U5RQUTJ6/20230918_185612.jpg", + "permalink_public": "https://slack-files.com/T01CWUYP5AR-F05U5RQUTJ6-56b64e8e34", + "is_starred": false, + "has_rich_preview": false, + "file_access": "visible" + } + ], + "upload": false, + "user": "U02LXF3HUN7", + "ts": "1695827956.140429", + "blocks": [ + { + "type": "rich_text", + "block_id": "6V1Z6", + "elements": [ + { + "type": "rich_text_section", + "elements": [ + { + "type": "text", + "text": "Meetup recap: Toronto Meetup @ Airflow Summit, September 18, 2023", + "style": { + "bold": true + } + }, + { + "type": "text", + "text": "\nIt was great to see so many members of our community at this event! I counted 32 total attendees, with all but a handful being first-timers.\n" + }, + { + "type": "text", + "text": "Topics included:", + "style": { + "bold": true + } + }, + { + "type": "text", + "text": "\n" + } + ] + }, + { + "type": "rich_text_list", + "elements": [ + { + "type": "rich_text_section", + "elements": [ + { + "type": "text", + "text": "Presentation on the history, architecture and roadmap of the project by " + }, + { + "type": "user", + "user_id": "U01DCLP0GU9" + }, + { + "type": "text", + "text": " and " + }, + { + "type": "user", + "user_id": "U01HNKK4XAM" + }, + { + "type": "text", + "text": " " + } + ] + }, + { + "type": "rich_text_section", + "elements": [ + { + "type": "text", + "text": "Discussion of OpenLineage support in " + }, + { + "type": "link", + "url": "https://marquezproject.ai/", + "text": "Marquez" + }, + { + "type": "text", + "text": " by " + }, + { + "type": "user", + "user_id": "U01DCMDFHBK" + }, + { + "type": "text", + "text": " " + } + ] + }, + { + "type": "rich_text_section", + "elements": [ + { + "type": "text", + "text": "Presentation by " + }, + { + "type": "text", + "text": "Ye Liu ", + "style": { + "bold": true + } + }, + { + "type": "text", + "text": "and " + }, + { + "type": "text", + "text": "Ivan Perepelitca ", + "style": { + "bold": true + } + }, + { + "type": "text", + "text": "from " + }, + { + "type": "link", + "url": "https://metaphor.io/", + "text": "Metaphor" + }, + { + "type": "text", + "text": ", the social platform for data, about their integration" + } + ] + }, + { + "type": "rich_text_section", + "elements": [ + { + "type": "text", + "text": "Presentation by " + }, + { + "type": "user", + "user_id": "U02MK6YNAQ5" + }, + { + "type": "text", + "text": " about the Spark integration" + } + ] + }, + { + "type": "rich_text_section", + "elements": [ + { + "type": "text", + "text": "Presentation by " + }, + { + "type": "user", + "user_id": "U01RA9B5GG2" + }, + { + "type": "text", + "text": " about the Apache Airflow Provider" + } + ] + } + ], + "style": "bullet", + "indent": 0, + "border": 0 + }, + { + "type": "rich_text_section", + "elements": [ + { + "type": "text", + "text": "Thanks to all the presenters and attendees with a shout out to " + }, + { + "type": "user", + "user_id": "U01HNKK4XAM" + }, + { + "type": "text", + "text": " for the help with organizing and day-of logistics, " + }, + { + "type": "user", + "user_id": "U02S6F54MAB" + }, + { + "type": "text", + "text": " for the help with set up/clean up, and " + }, + { + "type": "user", + "user_id": "U0323HG8C8H" + }, + { + "type": "text", + "text": " for the crucial assist with the signup sheet.\nThis was our first meetup in Toronto, and we learned some valuable lessons about planning events in new cities — the first and foremost being to ask for a pic of the building! " + }, + { + "type": "emoji", + "name": "slightly_smiling_face", + "unicode": "1f642" + }, + { + "type": "text", + "text": " But it seemed like folks were undeterred, and the space itself lived up to expectations.\nFor a recording and clips from the meetup, head over to our " + }, + { + "type": "link", + "url": "https://www.youtube.com/channel/UCRMLy4AaSw_ka-gNV9nl7VQ/", + "text": "YouTube channel" + }, + { + "type": "text", + "text": ".\n" + }, + { + "type": "text", + "text": "Upcoming events:", + "style": { + "bold": true + } + }, + { + "type": "text", + "text": "\n" + } + ] + }, + { + "type": "rich_text_list", + "elements": [ + { + "type": "rich_text_section", + "elements": [ + { + "type": "text", + "text": "October 5th in San Francisco: Marquez Meetup @ Astronomer (sign up " + }, + { + "type": "link", + "url": "https://www.meetup.com/meetup-group-bnfqymxe/events/295444209/?utm_medium=referral&utm_campaign=share-btn_savedevents_share_modal&utm_source=link", + "text": "here" + }, + { + "type": "text", + "text": ")" + } + ] + }, + { + "type": "rich_text_section", + "elements": [ + { + "type": "text", + "text": "November: Warsaw meetup (details, date TBA)" + } + ] + }, + { + "type": "rich_text_section", + "elements": [ + { + "type": "text", + "text": "January: London meetup (details, date TBA)" + } + ] + } + ], + "style": "bullet", + "indent": 0, + "border": 0 + }, + { + "type": "rich_text_section", + "elements": [ + { + "type": "text", + "text": "Are you interested in hosting or co-hosting an OpenLineage or Marquez meetup? DM me!" + } + ] + } + ] + } + ], + "client_msg_id": "fe688ab1-cced-4d37-9c89-e55c97ba1b41", + "thread_ts": "1695827956.140429", + "reply_count": 1, + "reply_users_count": 1, + "latest_reply": "1695830147.896309", + "reply_users": [ + "U02LXF3HUN7" + ], + "is_locked": false, + "subscribed": true, + "last_read": "1695830147.896309", + "attachments": [ + { + "image_url": "https://static.metaphor.io/preview.jpg", + "image_width": 719, + "image_height": 378, + "image_bytes": 122301, + "from_url": "https://metaphor.io/", + "id": 1, + "original_url": "https://metaphor.io/", + "fallback": "Metaphor - The Social Platform for Data", + "text": "Making Data Actionable, At Scale - Designed for data teams building cloud-native, self-service data platforms for their business users. Explore our Data Governance, Data Lineage, Data Discovery, and Data Trust capabilities today.", + "title": "Metaphor - The Social Platform for Data", + "title_link": "https://metaphor.io/", + "service_name": "metaphor.io" + }, + { + "from_url": "https://www.youtube.com/channel/UCRMLy4AaSw_ka-gNV9nl7VQ/", + "service_icon": "https://www.youtube.com/s/desktop/c0b97319/img/favicon.ico", + "thumb_url": "https://yt3.googleusercontent.com/6QCXvndRo0IJtiNvF9ZhZU5xPFkqvBFLORRgrzt4slwfEjYnbszvTNT41DFYhoESCd2Nb-29Lm0", + "thumb_width": 283, + "thumb_height": 283, + "id": 2, + "original_url": "https://www.youtube.com/channel/UCRMLy4AaSw_ka-gNV9nl7VQ/", + "fallback": "YouTube: OpenLineage Project", + "text": "Meetings, talks and tutorials by the OpenLineage Project, an Open Standard for lineage metadata collection", + "title": "OpenLineage Project", + "title_link": "https://www.youtube.com/channel/UCRMLy4AaSw_ka-gNV9nl7VQ/", + "service_name": "YouTube" + }, + { + "from_url": "https://www.meetup.com/meetup-group-bnfqymxe/events/295444209/?utm_medium=referral&utm_campaign=share-btn_savedevents_share_modal&utm_source=link", + "image_url": "https://secure.meetupstatic.com/photos/event/a/1/8/c/600_515141356.jpeg", + "image_width": 600, + "image_height": 338, + "image_bytes": 12395, + "service_icon": "https://secure.meetupstatic.com/next/images/general/m_swarm_120x120.png", + "id": 3, + "original_url": "https://www.meetup.com/meetup-group-bnfqymxe/events/295444209/?utm_medium=referral&utm_campaign=share-btn_savedevents_share_modal&utm_source=link", + "fallback": "Meetup: Marquez Meetup @ Astronomer, Thu, Oct 5, 2023, 5:30 PM | Meetup", + "text": "Join us on Thursday, October 5th, from 5:30-8:30 pm to learn about the Marquez project. Meet other members of the community, get tips on making the most of the latest impro", + "title": "Marquez Meetup @ Astronomer, Thu, Oct 5, 2023, 5:30 PM | Meetup", + "title_link": "https://www.meetup.com/meetup-group-bnfqymxe/events/295444209/?utm_medium=referral&utm_campaign=share-btn_savedevents_share_modal&utm_source=link", + "service_name": "Meetup" + } + ], + "reactions": [ + { + "name": "raised_hands", + "users": [ + "U01HVNU6A4C", + "U01HNKK4XAM", + "U02MK6YNAQ5" + ], + "count": 3 + }, + { + "name": "heart", + "users": [ + "U02S6F54MAB", + "U01HNKK4XAM", + "U05TU0U224A", + "U02MK6YNAQ5", + "U01DCLP0GU9", + "U01DCMDFHBK" + ], + "count": 6 + }, + { + "name": "rocket", + "users": [ + "U02S6F54MAB", + "U05Q3HT6PBR" + ], + "count": 2 + }, + { + "name": "sweat_smile", + "users": [ + "U01HNKK4XAM" + ], + "count": 1 + }, + { + "name": "white_check_mark", + "users": [ + "U0323HG8C8H" + ], + "count": 1 + } + ], + "replies": [ + { + "type": "message", + "text": "A few more pics:", + "files": [ + { + "id": "F05U6445VM1", + "created": 1695830018, + "timestamp": 1695830018, + "name": "IMG_5462.jpg", + "title": "IMG_5462.jpg", + "mimetype": "image/jpeg", + "filetype": "jpg", + "pretty_type": "JPEG", + "user": "U02LXF3HUN7", + "user_team": "T01CWUYP5AR", + "editable": false, + "size": 673416, + "mode": "hosted", + "is_external": false, + "external_type": "", + "is_public": true, + "public_url_shared": false, + "display_as_bot": false, + "username": "", + "url_private": "https://files.slack.com/files-pri/T01CWUYP5AR-F05U6445VM1/img_5462.jpg", + "url_private_download": "https://files.slack.com/files-pri/T01CWUYP5AR-F05U6445VM1/download/img_5462.jpg", + "media_display_type": "unknown", + "thumb_64": "https://files.slack.com/files-tmb/T01CWUYP5AR-F05U6445VM1-0fd1bf2d2b/img_5462_64.jpg", + "thumb_80": "https://files.slack.com/files-tmb/T01CWUYP5AR-F05U6445VM1-0fd1bf2d2b/img_5462_80.jpg", + "thumb_360": "https://files.slack.com/files-tmb/T01CWUYP5AR-F05U6445VM1-0fd1bf2d2b/img_5462_360.jpg", + "thumb_360_w": 360, + "thumb_360_h": 270, + "thumb_480": "https://files.slack.com/files-tmb/T01CWUYP5AR-F05U6445VM1-0fd1bf2d2b/img_5462_480.jpg", + "thumb_480_w": 480, + "thumb_480_h": 360, + "thumb_160": "https://files.slack.com/files-tmb/T01CWUYP5AR-F05U6445VM1-0fd1bf2d2b/img_5462_160.jpg", + "thumb_720": "https://files.slack.com/files-tmb/T01CWUYP5AR-F05U6445VM1-0fd1bf2d2b/img_5462_720.jpg", + "thumb_720_w": 720, + "thumb_720_h": 540, + "thumb_800": "https://files.slack.com/files-tmb/T01CWUYP5AR-F05U6445VM1-0fd1bf2d2b/img_5462_800.jpg", + "thumb_800_w": 800, + "thumb_800_h": 600, + "thumb_960": "https://files.slack.com/files-tmb/T01CWUYP5AR-F05U6445VM1-0fd1bf2d2b/img_5462_960.jpg", + "thumb_960_w": 960, + "thumb_960_h": 720, + "thumb_1024": "https://files.slack.com/files-tmb/T01CWUYP5AR-F05U6445VM1-0fd1bf2d2b/img_5462_1024.jpg", + "thumb_1024_w": 1024, + "thumb_1024_h": 768, + "original_w": 4032, + "original_h": 3024, + "thumb_tiny": "AwAkADCTAZlGO/8ASnGBDyMg+1O24lT8alAoWw2V/KkH3Xz9RQfMX70efoaskUhGRimIrgqwyKMY6GmqTsz2ApyYZQw71mWLLJtlXGPunr9aBOf9mop1aUAEgY9BUfk/IFLZGfSi4mkWvtB/2fzpPtHsv51U+zrjG4/nTo4hG25Sfxp3FyokbJh+X8aZG7IrNj5QOQKcxLIRyMj+Gq4WTaQQcenrRcdiySaciBupNMqaOpGL5SehP40gRR/CKkPSm0ANZRtPFRHhc+9TN9w1Cfuf8CoA/9k=", + "permalink": "https://openlineage.slack.com/files/U02LXF3HUN7/F05U6445VM1/img_5462.jpg", + "permalink_public": "https://slack-files.com/T01CWUYP5AR-F05U6445VM1-338a517e97", + "is_starred": false, + "has_rich_preview": false, + "file_access": "visible" + }, + { + "id": "F05U673KN1G", + "created": 1695830054, + "timestamp": 1695830054, + "name": "20230918_172756.jpg", + "title": "20230918_172756.jpg", + "mimetype": "image/jpeg", + "filetype": "jpg", + "pretty_type": "JPEG", + "user": "U02LXF3HUN7", + "user_team": "T01CWUYP5AR", + "editable": false, + "size": 2040975, + "mode": "hosted", + "is_external": false, + "external_type": "", + "is_public": true, + "public_url_shared": false, + "display_as_bot": false, + "username": "", + "url_private": "https://files.slack.com/files-pri/T01CWUYP5AR-F05U673KN1G/20230918_172756.jpg", + "url_private_download": "https://files.slack.com/files-pri/T01CWUYP5AR-F05U673KN1G/download/20230918_172756.jpg", + "media_display_type": "unknown", + "thumb_64": "https://files.slack.com/files-tmb/T01CWUYP5AR-F05U673KN1G-75177dd50c/20230918_172756_64.jpg", + "thumb_80": "https://files.slack.com/files-tmb/T01CWUYP5AR-F05U673KN1G-75177dd50c/20230918_172756_80.jpg", + "thumb_360": "https://files.slack.com/files-tmb/T01CWUYP5AR-F05U673KN1G-75177dd50c/20230918_172756_360.jpg", + "thumb_360_w": 360, + "thumb_360_h": 270, + "thumb_480": "https://files.slack.com/files-tmb/T01CWUYP5AR-F05U673KN1G-75177dd50c/20230918_172756_480.jpg", + "thumb_480_w": 480, + "thumb_480_h": 360, + "thumb_160": "https://files.slack.com/files-tmb/T01CWUYP5AR-F05U673KN1G-75177dd50c/20230918_172756_160.jpg", + "thumb_720": "https://files.slack.com/files-tmb/T01CWUYP5AR-F05U673KN1G-75177dd50c/20230918_172756_720.jpg", + "thumb_720_w": 720, + "thumb_720_h": 540, + "thumb_800": "https://files.slack.com/files-tmb/T01CWUYP5AR-F05U673KN1G-75177dd50c/20230918_172756_800.jpg", + "thumb_800_w": 800, + "thumb_800_h": 600, + "thumb_960": "https://files.slack.com/files-tmb/T01CWUYP5AR-F05U673KN1G-75177dd50c/20230918_172756_960.jpg", + "thumb_960_w": 960, + "thumb_960_h": 720, + "thumb_1024": "https://files.slack.com/files-tmb/T01CWUYP5AR-F05U673KN1G-75177dd50c/20230918_172756_1024.jpg", + "thumb_1024_w": 1024, + "thumb_1024_h": 768, + "original_w": 4032, + "original_h": 3024, + "thumb_tiny": "AwAkADB+KM1IACopmKErg3YOtL7dBRjJ/ClUhiQD0oegISl2kjNAA5Oc07B9KAIN0hfAIVR04zmkCsRwxNMkJEoA6mnRARS4JzkdfT2pFEqqV6H86hicuxIOPm5B7ippv9Udp5yKr2pIlbI4OfzoESkbckhVXdnr1pgldpg5JKg9B0HFPkiy+WySemDxUD5R2AY444p3AkaotxE20dKlNQn/AI+KhFEynB+tSRcF/rUa/eqSPq31piHYD5zx7iqB+8a0E/irPP3j9aYH/9k=", + "permalink": "https://openlineage.slack.com/files/U02LXF3HUN7/F05U673KN1G/20230918_172756.jpg", + "permalink_public": "https://slack-files.com/T01CWUYP5AR-F05U673KN1G-f3882acbd5", + "is_starred": false, + "has_rich_preview": false, + "file_access": "visible" + }, + { + "id": "F05U39M0MPX", + "created": 1695830139, + "timestamp": 1695830139, + "name": "20230918_172935.jpg", + "title": "20230918_172935.jpg", + "mimetype": "image/jpeg", + "filetype": "jpg", + "pretty_type": "JPEG", + "user": "U02LXF3HUN7", + "user_team": "T01CWUYP5AR", + "editable": false, + "size": 1734799, + "mode": "hosted", + "is_external": false, + "external_type": "", + "is_public": true, + "public_url_shared": false, + "display_as_bot": false, + "username": "", + "url_private": "https://files.slack.com/files-pri/T01CWUYP5AR-F05U39M0MPX/20230918_172935.jpg", + "url_private_download": "https://files.slack.com/files-pri/T01CWUYP5AR-F05U39M0MPX/download/20230918_172935.jpg", + "media_display_type": "unknown", + "thumb_64": "https://files.slack.com/files-tmb/T01CWUYP5AR-F05U39M0MPX-e9294fbbc2/20230918_172935_64.jpg", + "thumb_80": "https://files.slack.com/files-tmb/T01CWUYP5AR-F05U39M0MPX-e9294fbbc2/20230918_172935_80.jpg", + "thumb_360": "https://files.slack.com/files-tmb/T01CWUYP5AR-F05U39M0MPX-e9294fbbc2/20230918_172935_360.jpg", + "thumb_360_w": 360, + "thumb_360_h": 270, + "thumb_480": "https://files.slack.com/files-tmb/T01CWUYP5AR-F05U39M0MPX-e9294fbbc2/20230918_172935_480.jpg", + "thumb_480_w": 480, + "thumb_480_h": 360, + "thumb_160": "https://files.slack.com/files-tmb/T01CWUYP5AR-F05U39M0MPX-e9294fbbc2/20230918_172935_160.jpg", + "thumb_720": "https://files.slack.com/files-tmb/T01CWUYP5AR-F05U39M0MPX-e9294fbbc2/20230918_172935_720.jpg", + "thumb_720_w": 720, + "thumb_720_h": 540, + "thumb_800": "https://files.slack.com/files-tmb/T01CWUYP5AR-F05U39M0MPX-e9294fbbc2/20230918_172935_800.jpg", + "thumb_800_w": 800, + "thumb_800_h": 600, + "thumb_960": "https://files.slack.com/files-tmb/T01CWUYP5AR-F05U39M0MPX-e9294fbbc2/20230918_172935_960.jpg", + "thumb_960_w": 960, + "thumb_960_h": 720, + "thumb_1024": "https://files.slack.com/files-tmb/T01CWUYP5AR-F05U39M0MPX-e9294fbbc2/20230918_172935_1024.jpg", + "thumb_1024_w": 1024, + "thumb_1024_h": 768, + "original_w": 4032, + "original_h": 3024, + "thumb_tiny": "AwAkADCXy19KNnuRUq9KdVEEO3/aNRrkkjg4NWCoIqJBjePekykNxSFcjNOBBYrnkDOKKkockgCj6U7zh6GqY83AGVH4ZpSH7yt+AxTuKxaMvotNjOWf/Paq6rt5yW/3uad5mPalcLDYmk85u5wcikhcu5Y+gFNQlnd1NJbuRIVI60DJgeaZkk4pw603+KkBKEGO5ps4AgbHpUnao7j/AFDUwK1sxDN06VY3YPQflVa3+8fpVj0qkJn/2Q==", + "permalink": "https://openlineage.slack.com/files/U02LXF3HUN7/F05U39M0MPX/20230918_172935.jpg", + "permalink_public": "https://slack-files.com/T01CWUYP5AR-F05U39M0MPX-5aa7326d25", + "is_starred": false, + "has_rich_preview": false, + "file_access": "visible" + } + ], + "upload": false, + "user": "U02LXF3HUN7", + "ts": "1695830147.896309", + "blocks": [ + { + "type": "rich_text", + "block_id": "d56Cl", + "elements": [ + { + "type": "rich_text_section", + "elements": [ + { + "type": "text", + "text": "A few more pics:" + } + ] + } + ] + } + ], + "client_msg_id": "53cb1e19-536e-417d-8356-97b45bc1b35d", + "thread_ts": "1695827956.140429", + "parent_user_id": "U02LXF3HUN7" + } + ] + }, + { + "client_msg_id": "b0d8792e-684f-4d87-810a-a9f024db51f8", + "type": "message", + "text": " In Airflow Integration we send across a lineage Event for Dag start and complete, but that is not the case with spark integration…we don’t receive any event for the application start and complete in spark…is this expected behaviour or am i missing something?", + "user": "U05QL7LN2GH", + "ts": "1695703890.171789", + "blocks": [ + { + "type": "rich_text", + "block_id": "VWppi", + "elements": [ + { + "type": "rich_text_section", + "elements": [ + { + "type": "broadcast", + "range": "here" + }, + { + "type": "text", + "text": " In Airflow Integration we send across a lineage Event for Dag start and complete, but that is not the case with spark integration…we don’t receive any event for the application start and complete in spark…is this expected behaviour or am i missing something?" + } + ] + } + ] + } + ], + "team": "T01CWUYP5AR", + "thread_ts": "1695703890.171789", + "reply_count": 10, + "reply_users_count": 3, + "latest_reply": "1695822848.078309", + "reply_users": [ + "U02MK6YNAQ5", + "U05QL7LN2GH", + "U02LXF3HUN7" + ], + "is_locked": false, + "subscribed": true, + "last_read": "1695822848.078309", + "reactions": [ + { + "name": "heavy_plus_sign", + "users": [ + "U05A1D80QKF" + ], + "count": 1 + } + ], + "replies": [ + { + "client_msg_id": "8fc1de0f-13c7-45d4-a92e-9336dfe16d9e", + "type": "message", + "text": "For spark we do send `start` and `complete` for each spark action being run (single operation that causes spark processing being run). However, it is difficult for us to know if we're dealing with the last action within spark job or a spark script.", + "user": "U02MK6YNAQ5", + "ts": "1695822459.280709", + "blocks": [ + { + "type": "rich_text", + "block_id": "NnLoz", + "elements": [ + { + "type": "rich_text_section", + "elements": [ + { + "type": "text", + "text": "For spark we do send " + }, + { + "type": "text", + "text": "start", + "style": { + "code": true + } + }, + { + "type": "text", + "text": " and " + }, + { + "type": "text", + "text": "complete", + "style": { + "code": true + } + }, + { + "type": "text", + "text": " for each spark action being run (single operation that causes spark processing being run). However, it is difficult for us to know if we're dealing with the last action within spark job or a spark script." + } + ] + } + ] + } + ], + "team": "T01CWUYP5AR", + "thread_ts": "1695703890.171789", + "parent_user_id": "U05QL7LN2GH" + }, + { + "client_msg_id": "e4b03d27-5abf-4e74-a947-0ec863d48b6e", + "type": "message", + "text": "I think we need to look deeper into that as there is reoccuring need to capture such information", + "user": "U02MK6YNAQ5", + "ts": "1695822575.499259", + "blocks": [ + { + "type": "rich_text", + "block_id": "Uge+Y", + "elements": [ + { + "type": "rich_text_section", + "elements": [ + { + "type": "text", + "text": "I think we need to look deeper into that as there is reoccuring need to capture such information" + } + ] + } + ] + } + ], + "team": "T01CWUYP5AR", + "thread_ts": "1695703890.171789", + "parent_user_id": "U05QL7LN2GH" + }, + { + "client_msg_id": "c7181c17-e689-4a52-8d94-6adf7af1abad", + "type": "message", + "text": "and spark listener event has methods like `onApplicationStart` and `onApplicationEnd`", + "user": "U02MK6YNAQ5", + "ts": "1695822597.247609", + "blocks": [ + { + "type": "rich_text", + "block_id": "KzOiX", + "elements": [ + { + "type": "rich_text_section", + "elements": [ + { + "type": "text", + "text": "and spark listener event has methods like " + }, + { + "type": "text", + "text": "onApplicationStart", + "style": { + "code": true + } + }, + { + "type": "text", + "text": " and " + }, + { + "type": "text", + "text": "onApplicationEnd", + "style": { + "code": true + } + } + ] + } + ] + } + ], + "team": "T01CWUYP5AR", + "thread_ts": "1695703890.171789", + "parent_user_id": "U05QL7LN2GH" + }, + { + "client_msg_id": "b862a210-4843-4a38-8765-5f8289d97b5a", + "type": "message", + "text": "We are using the SparkListener, which has a function called OnApplicationStart which gets called whenever a spark application starts, so i was thinking why cant we send one at start and simlarly at end as well", + "user": "U05QL7LN2GH", + "ts": "1695822613.085959", + "blocks": [ + { + "type": "rich_text", + "block_id": "z28yv", + "elements": [ + { + "type": "rich_text_section", + "elements": [ + { + "type": "text", + "text": "We are using the SparkListener, which has a function called OnApplicationStart which gets called whenever a spark application starts, so i was thinking why cant we send one at start and simlarly at end as well" + } + ] + } + ] + } + ], + "team": "T01CWUYP5AR", + "thread_ts": "1695703890.171789", + "parent_user_id": "U05QL7LN2GH" + }, + { + "client_msg_id": "0564b6a2-8ee8-4943-a953-6ab6362a23c2", + "type": "message", + "text": "additionally, we would like to have a concept of a parent run for a spark job which aggregates all actions run within a single spark job context", + "user": "U02MK6YNAQ5", + "ts": "1695822633.475229", + "blocks": [ + { + "type": "rich_text", + "block_id": "B7f8D", + "elements": [ + { + "type": "rich_text_section", + "elements": [ + { + "type": "text", + "text": "additionally, we would like to have a concept of a parent run for a spark job which aggregates all actions run within a single spark job context" + } + ] + } + ] + } + ], + "team": "T01CWUYP5AR", + "thread_ts": "1695703890.171789", + "parent_user_id": "U05QL7LN2GH" + }, + { + "client_msg_id": "9c2cf2f2-80b8-4333-8590-d58f2e1b68b0", + "type": "message", + "text": "yeah exactly. the way that it works with airflow integration", + "user": "U05QL7LN2GH", + "ts": "1695822671.102329", + "blocks": [ + { + "type": "rich_text", + "block_id": "8Cgct", + "elements": [ + { + "type": "rich_text_section", + "elements": [ + { + "type": "text", + "text": "yeah exactly. the way that it works with airflow integration" + } + ] + } + ] + } + ], + "team": "T01CWUYP5AR", + "thread_ts": "1695703890.171789", + "parent_user_id": "U05QL7LN2GH" + }, + { + "client_msg_id": "1a09efa6-9259-4c58-8755-0d45fb51bf91", + "type": "message", + "text": "we do have an issue for that ", + "user": "U02MK6YNAQ5", + "ts": "1695822686.662719", + "blocks": [ + { + "type": "rich_text", + "block_id": "eVJKn", + "elements": [ + { + "type": "rich_text_section", + "elements": [ + { + "type": "text", + "text": "we do have an issue for that " + }, + { + "type": "link", + "url": "https://github.com/OpenLineage/OpenLineage/issues/2105" + } + ] + } + ] + } + ], + "team": "T01CWUYP5AR", + "attachments": [ + { + "id": 1, + "footer_icon": "https://slack.github.com/static/img/favicon-neutral.png", + "ts": 1694633153, + "color": "36a64f", + "bot_id": "B01VA0FB340", + "app_unfurl_url": "https://github.com/OpenLineage/OpenLineage/issues/2105", + "is_app_unfurl": true, + "app_id": "A01BP7R4KNY", + "fallback": "#2105 [PROPOSAL] Spark jobs should have a parent Job describing the Run State of Spark pipeline.", + "text": "*Purpose:* \nCurrently there is no way to group spark jobs running within the same SparkContext since there is no concept of parent Jobs in Spark. In airflow we support Parent Runs and this way we can group Tasks together within a single DAG.\n\n*Use cases:*\n\n• For data catalog based OL backends, a user might want to search for Spark ETL pipelines first and then dive deeper into a specific pipeline to view the Spark Jobs. This is similar to how a user can search for airflow DAG assets and then look into the Tasks for that DAG.\n• Also this will help display operational metadata for Spark pipelines. For Airflow we have DAG run status (parent job status), but for Spark we don't have any such concept to display the completion state of a pipeline run. We can only track it at SparkJob level.\n\n*Proposed implementation* \nThis should be similar to the airflow implementation.", + "title": "#2105 [PROPOSAL] Spark jobs should have a parent Job describing the Run State of Spark pipeline.", + "title_link": "https://github.com/OpenLineage/OpenLineage/issues/2105", + "footer": "", + "fields": [ + { + "value": "proposal", + "title": "Labels", + "short": true + }, + { + "value": "2", + "title": "Comments", + "short": true + } + ], + "mrkdwn_in": [ + "text" + ] + } + ], + "thread_ts": "1695703890.171789", + "parent_user_id": "U05QL7LN2GH" + }, + { + "client_msg_id": "9400483a-7331-4225-b6fc-ec10a8512e76", + "type": "message", + "text": "what you can is: come to our monthly Openlineage open meetings and raise that issue and convince the community about its importance", + "user": "U02MK6YNAQ5", + "ts": "1695822728.304729", + "blocks": [ + { + "type": "rich_text", + "block_id": "VDW1M", + "elements": [ + { + "type": "rich_text_section", + "elements": [ + { + "type": "text", + "text": "what you can is: come to our monthly Openlineage open meetings and raise that issue and convince the community about its importance" + } + ] + } + ] + } + ], + "team": "T01CWUYP5AR", + "thread_ts": "1695703890.171789", + "parent_user_id": "U05QL7LN2GH" + }, + { + "client_msg_id": "de5ac6ca-b957-4e76-93dd-25d7c2e0e2b2", + "type": "message", + "text": "yeah sure would love to do that…how can i join them, will that be posted here in this slack channel?", + "user": "U05QL7LN2GH", + "ts": "1695822812.568049", + "blocks": [ + { + "type": "rich_text", + "block_id": "hrUZs", + "elements": [ + { + "type": "rich_text_section", + "elements": [ + { + "type": "text", + "text": "yeah sure would love to do that…how can i join them, will that be posted here in this slack channel?" + } + ] + } + ] + } + ], + "team": "T01CWUYP5AR", + "thread_ts": "1695703890.171789", + "parent_user_id": "U05QL7LN2GH" + }, + { + "client_msg_id": "08fddf36-1931-427e-b16b-cd95cae49c05", + "type": "message", + "text": "Hi, you can see the schedule and RSVP here: ", + "user": "U02LXF3HUN7", + "ts": "1695822848.078309", + "blocks": [ + { + "type": "rich_text", + "block_id": "ASLvc", + "elements": [ + { + "type": "rich_text_section", + "elements": [ + { + "type": "text", + "text": "Hi, you can see the schedule and RSVP here: " + }, + { + "type": "link", + "url": "https://openlineage.io/community" + } + ] + } + ] + } + ], + "team": "T01CWUYP5AR", + "attachments": [ + { + "from_url": "https://openlineage.io/community", + "service_icon": "https://openlineage.io/img/favicon.ico", + "id": 1, + "original_url": "https://openlineage.io/community", + "fallback": "Community | OpenLineage", + "text": "Data lineage is the foundation for a new generation of powerful, context-aware data tools and best practices. OpenLineage enables consistent collection of lineage metadata, creating a deeper understanding of how data is produced and used.", + "title": "Community | OpenLineage", + "title_link": "https://openlineage.io/community", + "service_name": "openlineage.io" + } + ], + "thread_ts": "1695703890.171789", + "parent_user_id": "U05QL7LN2GH", + "reactions": [ + { + "name": "raised_hands", + "users": [ + "U02MK6YNAQ5" + ], + "count": 1 + }, + { + "name": "gratitude-thank-you", + "users": [ + "U05QL7LN2GH" + ], + "count": 1 + } + ] + } + ] + }, + { + "client_msg_id": "8a4ccf7e-640f-44e0-9d4d-92bcbd341376", + "type": "message", + "text": "I'm using the Spark OpenLineage integration. In the `outputStatistics` output dataset facet we receive `rowCount` and `size`.\nThe Job performs a SQL insert into a MySQL table and I'm receiving the `size` as 0.\n```{\n \"outputStatistics\":\n {\n \"_producer\": \"\",\n \"_schemaURL\": \"\",\n \"rowCount\": 1,\n \"size\": 0\n }\n}```\nI'm not sure what the size means here. Does this mean number of bytes inserted/updated?\nAlso, do we have any documentation for Spark specific Job and Run facets?", + "user": "U05A1D80QKF", + "ts": "1695663385.834539", + "blocks": [ + { + "type": "rich_text", + "block_id": "JoEAD", + "elements": [ + { + "type": "rich_text_section", + "elements": [ + { + "type": "text", + "text": "I'm using the Spark OpenLineage integration. In the " + }, + { + "type": "text", + "text": "outputStatistics", + "style": { + "code": true + } + }, + { + "type": "text", + "text": " output dataset facet we receive " + }, + { + "type": "text", + "text": "rowCount", + "style": { + "code": true + } + }, + { + "type": "text", + "text": " and " + }, + { + "type": "text", + "text": "size", + "style": { + "code": true + } + }, + { + "type": "text", + "text": ".\nThe Job performs a SQL insert into a MySQL table and I'm receiving the " + }, + { + "type": "text", + "text": "size", + "style": { + "code": true + } + }, + { + "type": "text", + "text": " as 0.\n" + } + ] + }, + { + "type": "rich_text_preformatted", + "elements": [ + { + "type": "text", + "text": "{\n \"outputStatistics\":\n {\n \"_producer\": \"" + }, + { + "type": "link", + "url": "https://github.com/OpenLineage/OpenLineage/tree/1.1.0/integration/spark" + }, + { + "type": "text", + "text": "\",\n \"_schemaURL\": \"" + }, + { + "type": "link", + "url": "https://openlineage.io/spec/facets/1-0-0/OutputStatisticsOutputDatasetFacet.json#/$defs/OutputStatisticsOutputDatasetFacet" + }, + { + "type": "text", + "text": "\",\n \"rowCount\": 1,\n \"size\": 0\n }\n}" + } + ], + "border": 0 + }, + { + "type": "rich_text_section", + "elements": [ + { + "type": "text", + "text": "I'm not sure what the size means here. Does this mean number of bytes inserted/updated?\nAlso, do we have any documentation for Spark specific Job and Run facets?" + } + ] + } + ] + } + ], + "team": "T01CWUYP5AR", + "edited": { + "user": "U05A1D80QKF", + "ts": "1695663417.000000" + }, + "thread_ts": "1695663385.834539", + "reply_count": 1, + "reply_users_count": 1, + "latest_reply": "1695822960.410559", + "reply_users": [ + "U02MK6YNAQ5" + ], + "is_locked": false, + "subscribed": false, + "replies": [ + { + "client_msg_id": "15851c75-f1f1-4575-b4ba-0444b0ae475b", + "type": "message", + "text": "I am not sure it's stated in the doc. Here's the list of spark facets schemas: ", + "user": "U02MK6YNAQ5", + "ts": "1695822960.410559", + "blocks": [ + { + "type": "rich_text", + "block_id": "WlBQw", + "elements": [ + { + "type": "rich_text_section", + "elements": [ + { + "type": "text", + "text": "I am not sure it's stated in the doc. Here's the list of spark facets schemas: " + }, + { + "type": "link", + "url": "https://github.com/OpenLineage/OpenLineage/tree/main/integration/spark/shared/facets/spark/v1" + } + ] + } + ] + } + ], + "team": "T01CWUYP5AR", + "thread_ts": "1695663385.834539", + "parent_user_id": "U05A1D80QKF" + } + ] + }, + { + "client_msg_id": "05bcc77f-ca23-4be3-ae32-e73df9d4cce9", + "type": "message", + "text": " I'm presently addressing a particular scenario that pertains to Openlineage authentication, specifically involving the use of an access key and secret.\n\nI've implemented a custom token provider called AccessKeySecretKeyTokenProvider, which extends the TokenProvider class. This token provider communicates with another service, obtaining a token and an expiration time based on the provided access key, secret, and client ID.\n\nMy goal is to retain this token in a cache prior to its expiration, thereby eliminating the need for network calls to the third-party service. Is it possible without relying on an external caching system.", + "user": "U05SMTVPPL3", + "ts": "1695633110.066819", + "blocks": [ + { + "type": "rich_text", + "block_id": "Y8xCm", + "elements": [ + { + "type": "rich_text_section", + "elements": [ + { + "type": "broadcast", + "range": "here" + }, + { + "type": "text", + "text": " I'm presently addressing a particular scenario that pertains to Openlineage authentication, specifically involving the use of an access key and secret.\n\nI've implemented a custom token provider called AccessKeySecretKeyTokenProvider, which extends the TokenProvider class. This token provider communicates with another service, obtaining a token and an expiration time based on the provided access key, secret, and client ID.\n\nMy goal is to retain this token in a cache prior to its expiration, thereby eliminating the need for network calls to the third-party service. Is it possible without relying on an external caching system." + } + ] + } + ] + } + ], + "team": "T01CWUYP5AR", + "edited": { + "user": "U05SMTVPPL3", + "ts": "1695642342.000000" + }, + "thread_ts": "1695633110.066819", + "reply_count": 11, + "reply_users_count": 3, + "latest_reply": "1695716393.265469", + "reply_users": [ + "U01HNKK4XAM", + "U05SMTVPPL3", + "U02MK6YNAQ5" + ], + "is_locked": false, + "subscribed": false, + "replies": [ + { + "client_msg_id": "910899d6-33f1-49ac-bd72-82521a106ad5", + "type": "message", + "text": "Hey <@U05SMTVPPL3>, I’m not sure that I fully understand your question here. What do you mean by OpenLineage authentication?\nWhat are you using to generate OL events? What’s your OL receiving backend?", + "user": "U01HNKK4XAM", + "ts": "1695646613.043569", + "blocks": [ + { + "type": "rich_text", + "block_id": "Mf6c0", + "elements": [ + { + "type": "rich_text_section", + "elements": [ + { + "type": "text", + "text": "Hey " + }, + { + "type": "user", + "user_id": "U05SMTVPPL3" + }, + { + "type": "text", + "text": ", I’m not sure that I fully understand your question here. What do you mean by OpenLineage authentication?\nWhat are you using to generate OL events? What’s your OL receiving backend?" + } + ] + } + ] + } + ], + "team": "T01CWUYP5AR", + "thread_ts": "1695633110.066819", + "parent_user_id": "U05SMTVPPL3" + }, + { + "client_msg_id": "d42d6001-6201-4fab-810d-5bd57f54ffd2", + "type": "message", + "text": "Hey <@U01HNKK4XAM>,\nI wanted to clarify the previous message. I apologize for any confusion. When I mentioned \"_OpenLineage authentication_,\" I was actually referring to the authentication process for the OpenLineage backend, specifically using HTTP transport. This involves using my custom token provider, which utilizes access keys and secrets for authentication. The OL backend is http based backend . I hope this clears things up!", + "user": "U05SMTVPPL3", + "ts": "1695647073.016419", + "blocks": [ + { + "type": "rich_text", + "block_id": "usb+A", + "elements": [ + { + "type": "rich_text_section", + "elements": [ + { + "type": "text", + "text": "Hey " + }, + { + "type": "user", + "user_id": "U01HNKK4XAM" + }, + { + "type": "text", + "text": ",\nI wanted to clarify the previous message. I apologize for any confusion. When I mentioned \"" + }, + { + "type": "text", + "text": "OpenLineage authentication", + "style": { + "italic": true + } + }, + { + "type": "text", + "text": ",\" I was actually referring to the authentication process for the OpenLineage backend, specifically using HTTP transport. This involves using my custom token provider, which utilizes access keys and secrets for authentication. The OL backend is http based backend . I hope this clears things up!" + } + ] + } + ] + } + ], + "team": "T01CWUYP5AR", + "edited": { + "user": "U05SMTVPPL3", + "ts": "1695647102.000000" + }, + "thread_ts": "1695633110.066819", + "parent_user_id": "U05SMTVPPL3" + }, + { + "client_msg_id": "c6059924-e40d-490b-800e-2d91d960af05", + "type": "message", + "text": "Are you using Marquez?", + "user": "U01HNKK4XAM", + "ts": "1695647112.882869", + "blocks": [ + { + "type": "rich_text", + "block_id": "OByZ5", + "elements": [ + { + "type": "rich_text_section", + "elements": [ + { + "type": "text", + "text": "Are you using Marquez?" + } + ] + } + ] + } + ], + "team": "T01CWUYP5AR", + "thread_ts": "1695633110.066819", + "parent_user_id": "U05SMTVPPL3" + }, + { + "client_msg_id": "abffa9ed-8e4d-448b-91e7-36c2f86e8d06", + "type": "message", + "text": "We are trying to leverage our own backend here.", + "user": "U05SMTVPPL3", + "ts": "1695647155.937279", + "blocks": [ + { + "type": "rich_text", + "block_id": "sukxf", + "elements": [ + { + "type": "rich_text_section", + "elements": [ + { + "type": "text", + "text": "We are trying to leverage our own backend here." + } + ] + } + ] + } + ], + "team": "T01CWUYP5AR", + "thread_ts": "1695633110.066819", + "parent_user_id": "U05SMTVPPL3" + }, + { + "client_msg_id": "280b8e75-9356-4f23-bd2d-a897f6a2d41e", + "type": "message", + "text": "I see.. I’m not sure the OpenLineage community could help here. Which webserver framework are you using?", + "user": "U01HNKK4XAM", + "ts": "1695647223.932189", + "blocks": [ + { + "type": "rich_text", + "block_id": "5a6eW", + "elements": [ + { + "type": "rich_text_section", + "elements": [ + { + "type": "text", + "text": "I see.. I’m not sure the OpenLineage community could help here. Which webserver framework are you using?" + } + ] + } + ] + } + ], + "team": "T01CWUYP5AR", + "thread_ts": "1695633110.066819", + "parent_user_id": "U05SMTVPPL3" + }, + { + "client_msg_id": "d0948a0c-3d8a-47ba-bc3f-09e8322635fc", + "type": "message", + "text": "KTOR framework", + "user": "U05SMTVPPL3", + "ts": "1695647336.172679", + "blocks": [ + { + "type": "rich_text", + "block_id": "2+7aa", + "elements": [ + { + "type": "rich_text_section", + "elements": [ + { + "type": "text", + "text": "KTOR framework" + } + ] + } + ] + } + ], + "team": "T01CWUYP5AR", + "thread_ts": "1695633110.066819", + "parent_user_id": "U05SMTVPPL3" + }, + { + "client_msg_id": "bdf944a5-2fc7-4728-9cc3-23558c2acf34", + "type": "message", + "text": "Our backend authentication operates based on either a pair of keys or a single bearer token, with a limited time of expiry. Hence, wanted to cache this information inside the token provider.", + "user": "U05SMTVPPL3", + "ts": "1695647733.493489", + "blocks": [ + { + "type": "rich_text", + "block_id": "Vs7hy", + "elements": [ + { + "type": "rich_text_section", + "elements": [ + { + "type": "text", + "text": "Our backend authentication operates based on either a pair of keys or a single bearer token, with a limited time of expiry. Hence, wanted to cache this information inside the token provider." + } + ] + } + ] + } + ], + "team": "T01CWUYP5AR", + "thread_ts": "1695633110.066819", + "parent_user_id": "U05SMTVPPL3" + }, + { + "client_msg_id": "4cb9780f-caa0-48ec-a7c6-276b37dd3966", + "type": "message", + "text": "I see, I would ask this question here ", + "user": "U01HNKK4XAM", + "ts": "1695648417.273519", + "blocks": [ + { + "type": "rich_text", + "block_id": "QebU1", + "elements": [ + { + "type": "rich_text_section", + "elements": [ + { + "type": "text", + "text": "I see, I would ask this question here " + }, + { + "type": "link", + "url": "https://ktor.io/support/" + } + ] + } + ] + } + ], + "team": "T01CWUYP5AR", + "attachments": [ + { + "image_url": "https://ktor.io/static/preview-bcb70be2e64dd985ee1d3e7044b41726.png", + "image_width": 1280, + "image_height": 800, + "image_bytes": 313772, + "from_url": "https://ktor.io/support/", + "service_icon": "https://ktor.io/icons/icon-48x48.png?v=1c13dbdf035874784ecf822f992c1e4b", + "id": 1, + "original_url": "https://ktor.io/support/", + "fallback": "Ktor Framework: Support", + "text": "Kotlin Server and Client Framework for microservices, HTTP APIs, and RESTful services", + "title": "Support", + "title_link": "https://ktor.io/support/", + "service_name": "Ktor Framework" + } + ], + "thread_ts": "1695633110.066819", + "parent_user_id": "U05SMTVPPL3" + }, + { + "client_msg_id": "b0df4659-713a-428a-9a80-80d8b11fa670", + "type": "message", + "text": "Thank you", + "user": "U05SMTVPPL3", + "ts": "1695651172.772709", + "blocks": [ + { + "type": "rich_text", + "block_id": "ztrU8", + "elements": [ + { + "type": "rich_text_section", + "elements": [ + { + "type": "text", + "text": "Thank you" + } + ] + } + ] + } + ], + "team": "T01CWUYP5AR", + "thread_ts": "1695633110.066819", + "parent_user_id": "U05SMTVPPL3" + }, + { + "client_msg_id": "48ff099b-571d-4942-bcf6-18283df58da7", + "type": "message", + "text": "<@U05SMTVPPL3> which openlineage client are you using: java or python?", + "user": "U02MK6YNAQ5", + "ts": "1695716000.284039", + "blocks": [ + { + "type": "rich_text", + "block_id": "+TXCP", + "elements": [ + { + "type": "rich_text_section", + "elements": [ + { + "type": "user", + "user_id": "U05SMTVPPL3" + }, + { + "type": "text", + "text": " which openlineage client are you using: java or python?" + } + ] + } + ] + } + ], + "team": "T01CWUYP5AR", + "thread_ts": "1695633110.066819", + "parent_user_id": "U05SMTVPPL3" + }, + { + "client_msg_id": "70e67eba-ff5a-4653-bde6-13cc71bff642", + "type": "message", + "text": "<@U02MK6YNAQ5> I am using python client", + "user": "U05SMTVPPL3", + "ts": "1695716393.265469", + "blocks": [ + { + "type": "rich_text", + "block_id": "BID0d", + "elements": [ + { + "type": "rich_text_section", + "elements": [ + { + "type": "user", + "user_id": "U02MK6YNAQ5" + }, + { + "type": "text", + "text": " I am using python client" + } + ] + } + ] + } + ], + "team": "T01CWUYP5AR", + "thread_ts": "1695633110.066819", + "parent_user_id": "U05SMTVPPL3" + } + ] + }, + { + "type": "message", + "text": "I am attaching the log4j, there is no openlineagecontext", + "files": [ + { + "id": "F05TJ6PA3NG", + "created": 1695352480, + "timestamp": 1695352480, + "name": "log4j-active (3).txt", + "title": "log4j-active (3).txt", + "mimetype": "text/plain", + "filetype": "text", + "pretty_type": "Plain Text", + "user": "U05T8BJD4DU", + "user_team": "T01CWUYP5AR", + "editable": true, + "size": 229901, + "mode": "snippet", + "is_external": false, + "external_type": "", + "is_public": true, + "public_url_shared": false, + "display_as_bot": false, + "username": "", + "url_private": "https://files.slack.com/files-pri/T01CWUYP5AR-F05TJ6PA3NG/log4j-active__3_.txt", + "url_private_download": "https://files.slack.com/files-pri/T01CWUYP5AR-F05TJ6PA3NG/download/log4j-active__3_.txt", + "permalink": "https://openlineage.slack.com/files/U05T8BJD4DU/F05TJ6PA3NG/log4j-active__3_.txt", + "permalink_public": "https://slack-files.com/T01CWUYP5AR-F05TJ6PA3NG-030be021d8", + "edit_link": "https://openlineage.slack.com/files/U05T8BJD4DU/F05TJ6PA3NG/log4j-active__3_.txt/edit", + "preview": "23/09/22 03:12:03 INFO DriverDaemon$: Started Log4j2\n23/09/22 03:12:06 INFO DatabricksMain$$anon$1: Configured feature flag data source LaunchDarkly\n23/09/22 03:12:06 INFO DatabricksMain$$anon$1: Load feature flag from LaunchDarkly\n23/09/22 03:12:06 WARN DatabricksMain$$anon$1: REGION environment variable is not defined. getConfForCurrentRegion will always return default value\n23/09/22 03:12:06 INFO DriverDaemon$: Current JVM Version 1.8.0_362", + "preview_highlight": "
\n
\n
23/09/22 03:12:03 INFO DriverDaemon$: Started Log4j2
\n
23/09/22 03:12:06 INFO DatabricksMain$$anon$1: Configured feature flag data source LaunchDarkly
\n
23/09/22 03:12:06 INFO DatabricksMain$$anon$1: Load feature flag from LaunchDarkly
\n
23/09/22 03:12:06 WARN DatabricksMain$$anon$1: REGION environment variable is not defined. getConfForCurrentRegion will always return default value
\n
23/09/22 03:12:06 INFO DriverDaemon$: Current JVM Version 1.8.0_362
\n
\n
\n", + "lines": 1327, + "lines_more": 1322, + "preview_is_truncated": true, + "is_starred": false, + "has_rich_preview": false, + "file_access": "visible" + } + ], + "upload": false, + "user": "U05T8BJD4DU", + "display_as_bot": false, + "ts": "1695352570.560639", + "blocks": [ + { + "type": "rich_text", + "block_id": "+0tED", + "elements": [ + { + "type": "rich_text_section", + "elements": [ + { + "type": "text", + "text": "I am attaching the log4j, there is no openlineagecontext" + } + ] + } + ] + } + ], + "edited": { + "user": "U05T8BJD4DU", + "ts": "1695352579.000000" + }, + "client_msg_id": "a2763503-de63-4097-81ca-cb8eb43dace3", + "thread_ts": "1695352570.560639", + "reply_count": 2, + "reply_users_count": 2, + "latest_reply": "1695646750.596629", + "reply_users": [ + "U05T8BJD4DU", + "U01HNKK4XAM" + ], + "is_locked": false, + "subscribed": false, + "reactions": [ + { + "name": "white_check_mark", + "users": [ + "U01HNKK4XAM" + ], + "count": 1 + } + ], + "replies": [ + { + "client_msg_id": "6684ada9-9870-4370-95a1-0d17db24a2e9", + "type": "message", + "text": "this issue is resolved, solution can be found here: ", + "user": "U05T8BJD4DU", + "ts": "1695354442.264569", + "blocks": [ + { + "type": "rich_text", + "block_id": "nHzgi", + "elements": [ + { + "type": "rich_text_section", + "elements": [ + { + "type": "text", + "text": "this issue is resolved, solution can be found here: " + }, + { + "type": "link", + "url": "https://openlineage.slack.com/archives/C01CK9T7HKR/p1691592987038929" + } + ] + } + ] + } + ], + "team": "T01CWUYP5AR", + "attachments": [ + { + "from_url": "https://openlineage.slack.com/archives/C01CK9T7HKR/p1691592987038929", + "ts": "1691592987.038929", + "author_id": "U05KNSP01TR", + "channel_id": "C01CK9T7HKR", + "channel_team": "T01CWUYP5AR", + "is_msg_unfurl": true, + "is_thread_root_unfurl": true, + "message_blocks": [ + { + "team": "T01CWUYP5AR", + "channel": "C01CK9T7HKR", + "ts": "1691592987.038929", + "message": { + "blocks": [ + { + "type": "rich_text", + "block_id": "oRms", + "elements": [ + { + "type": "rich_text_section", + "elements": [ + { + "type": "text", + "text": "Hey,\nI’m running Spark application (spark version 3.4) with OL integration.\nI changed spark to use “debug” level, and I see the OL events with the below message:\n“Emitting lineage completed successfully:”\n\nWith all the above, I can’t see the event in Marquez.\n\nAttaching the OL configurations.\nWhen changing the OL-spark version to 0.6.+, I do see event created in Marquez with only “Start” status (attached below).\n\nThe OL-spark version is matching the Spark version? Is there a known issues with the Spark / OL versions ?" + } + ] + } + ] + } + ] + } + } + ], + "files": [ + { + "id": "F05M1RLTMKM", + "created": 1691592859, + "timestamp": 1691592859, + "user": "U05KNSP01TR", + "is_hidden_by_limit": 1 + }, + { + "id": "F05M1UZA5EW", + "created": 1691592943, + "timestamp": 1691592943, + "user": "U05KNSP01TR", + "is_hidden_by_limit": 1 + } + ], + "id": 1, + "original_url": "https://openlineage.slack.com/archives/C01CK9T7HKR/p1691592987038929", + "fallback": "[August 9th, 2023 7:56 AM] zahi.fail: Hey,\nI’m running Spark application (spark version 3.4) with OL integration.\nI changed spark to use “debug” level, and I see the OL events with the below message:\n“Emitting lineage completed successfully:”\n\nWith all the above, I can’t see the event in Marquez.\n\nAttaching the OL configurations.\nWhen changing the OL-spark version to 0.6.+, I do see event created in Marquez with only “Start” status (attached below).\n\nThe OL-spark version is matching the Spark version? Is there a known issues with the Spark / OL versions ?", + "text": "Hey,\nI’m running Spark application (spark version 3.4) with OL integration.\nI changed spark to use “debug” level, and I see the OL events with the below message:\n“Emitting lineage completed successfully:”\n\nWith all the above, I can’t see the event in Marquez.\n\nAttaching the OL configurations.\nWhen changing the OL-spark version to 0.6.+, I do see event created in Marquez with only “Start” status (attached below).\n\nThe OL-spark version is matching the Spark version? Is there a known issues with the Spark / OL versions ?", + "author_name": "Zahi Fail", + "author_link": "https://openlineage.slack.com/team/U05KNSP01TR", + "author_icon": "https://avatars.slack-edge.com/2023-08-03/5668907056247_deaa92eded7196343d84_48.png", + "author_subname": "Zahi Fail", + "mrkdwn_in": [ + "text" + ], + "footer": "Thread in Slack Conversation" + } + ], + "thread_ts": "1695352570.560639", + "parent_user_id": "U05T8BJD4DU" + }, + { + "client_msg_id": "8f1c101c-fcc5-40be-8178-e9254c71ed5f", + "type": "message", + "text": "We were all out at Airflow Summit last week, so apologies for the delayed response. Glad you were able to resolve the issue!", + "user": "U01HNKK4XAM", + "ts": "1695646750.596629", + "blocks": [ + { + "type": "rich_text", + "block_id": "i6p6k", + "elements": [ + { + "type": "rich_text_section", + "elements": [ + { + "type": "text", + "text": "We were all out at Airflow Summit last week, so apologies for the delayed response. Glad you were able to resolve the issue!" + } + ] + } + ] + } + ], + "team": "T01CWUYP5AR", + "thread_ts": "1695352570.560639", + "parent_user_id": "U05T8BJD4DU" + } + ] + }, + { + "client_msg_id": "00cbf211-1bbc-4bd3-8275-8cf7bea8013e", + "type": "message", + "text": "I installed 1.2.2 on Databricks, followed the below init script: \n\nmy cluster config looks like this:\n\nspark.openlineage.version v1\nspark.openlineage.namespace adb-5445974573286168.8#default\nspark.openlineage.endpoint v1/lineage\nspark.openlineage.url.param.code 8kZl0bo2TJfnbpFxBv-R2v7xBDj-PgWMol3yUm5iP1vaAzFu9kIZGg==\nspark.openlineage.url \n\nBut it is not calling the API, it works fine with 0.18 version", + "user": "U05T8BJD4DU", + "ts": "1695347501.889769", + "blocks": [ + { + "type": "rich_text", + "block_id": "6uik+", + "elements": [ + { + "type": "rich_text_section", + "elements": [ + { + "type": "text", + "text": "I installed 1.2.2 on Databricks, followed the below init script: " + }, + { + "type": "link", + "url": "https://github.com/OpenLineage/OpenLineage/blob/main/integration/spark/databricks/open-lineage-init-script.sh" + }, + { + "type": "text", + "text": "\n\nmy cluster config looks like this:\n\nspark.openlineage.version v1\nspark.openlineage.namespace adb-5445974573286168.8#default\nspark.openlineage.endpoint v1/lineage\nspark.openlineage.url.param.code 8kZl0bo2TJfnbpFxBv-R2v7xBDj-PgWMol3yUm5iP1vaAzFu9kIZGg==\nspark.openlineage.url " + }, + { + "type": "link", + "url": "https://f77b-50-35-69-138.ngrok-free.app" + }, + { + "type": "text", + "text": "\n\nBut it is not calling the API, it works fine with 0.18 version" + } + ] + } + ] + } + ], + "team": "T01CWUYP5AR", + "attachments": [ + { + "id": 1, + "footer_icon": "https://slack.github.com/static/img/favicon-neutral.png", + "color": "24292f", + "bot_id": "B01VA0FB340", + "app_unfurl_url": "https://github.com/OpenLineage/OpenLineage/blob/main/integration/spark/databricks/open-lineage-init-script.sh", + "is_app_unfurl": true, + "app_id": "A01BP7R4KNY", + "fallback": "", + "text": "```\n#!/bin/bash\n#\n# Copyright 2018-2023 contributors to the OpenLineage project\n# SPDX-License-Identifier: Apache-2.0\n\nSTAGE_DIR=\"/dbfs/databricks/openlineage\"\n\necho \"BEGIN: Upload Spark Listener JARs\"\ncp -f $STAGE_DIR/openlineage-spark-*.jar /mnt/driver-daemon/jars || { echo \"Error copying Spark Listener library file\"; exit 1;}\necho \"END: Upload Spark Listener JARs\"\n\necho \"BEGIN: Modify Spark config settings\"\ncat << 'EOF' > /databricks/driver/conf/openlineage-spark-driver-defaults.conf\n[driver] {\n \"spark.extraListeners\" = \"io.openlineage.spark.agent.OpenLineageSparkListener\"\n}\nEOF\necho \"END: Modify Spark config settings\"\n\n```", + "title": "", + "footer": "", + "mrkdwn_in": [ + "text" + ] + } + ], + "reactions": [ + { + "name": "white_check_mark", + "users": [ + "U01HNKK4XAM" + ], + "count": 1 + } + ] + }, + { + "client_msg_id": "ceb38ae7-a2ab-4d48-a920-f6e6d6c51b93", + "type": "message", + "text": "I am using this accelerator that leverages OpenLineage on Databricks to publish lineage info to Purview, but it's using a rather old version of OpenLineage aka 0.18, anybody has tried it on a newer version of OpenLineage? I am facing some issues with the inputs and outputs for the same object is having different json\n", + "user": "U05T8BJD4DU", + "ts": "1695335777.852519", + "blocks": [ + { + "type": "rich_text", + "block_id": "6JEze", + "elements": [ + { + "type": "rich_text_section", + "elements": [ + { + "type": "text", + "text": "I am using this accelerator that leverages OpenLineage on Databricks to publish lineage info to Purview, but it's using a rather old version of OpenLineage aka 0.18, anybody has tried it on a newer version of OpenLineage? I am facing some issues with the inputs and outputs for the same object is having different json\n" + }, + { + "type": "link", + "url": "https://github.com/microsoft/Purview-ADB-Lineage-Solution-Accelerator/", + "text": "https://github.com/microsoft/Purview-ADB-Lineage-Solution-Accelerator/" + } + ] + } + ] + } + ], + "team": "T01CWUYP5AR", + "edited": { + "user": "U05T8BJD4DU", + "ts": "1695335813.000000" + }, + "attachments": [ + { + "id": 1, + "color": "24292f", + "bot_id": "B01VA0FB340", + "app_unfurl_url": "https://github.com/microsoft/Purview-ADB-Lineage-Solution-Accelerator/", + "is_app_unfurl": true, + "app_id": "A01BP7R4KNY", + "fallback": "microsoft/Purview-ADB-Lineage-Solution-Accelerator", + "text": "A connector to ingest Azure Databricks lineage into Microsoft Purview", + "title": "microsoft/Purview-ADB-Lineage-Solution-Accelerator", + "fields": [ + { + "value": "77", + "title": "Stars", + "short": true + }, + { + "value": "C#", + "title": "Language", + "short": true + } + ] + } + ], + "reactions": [ + { + "name": "white_check_mark", + "users": [ + "U01HNKK4XAM" + ], + "count": 1 + } + ] + }, + { + "client_msg_id": "6fdd1e8f-8f13-4100-9ea4-1af90dd34bc8", + "type": "message", + "text": "Hi, we're using Custom Operators in airflow(2.5) and are planning to expose lineage via default extractors: \n*Question*: Now if we upgrade our Airflow version to 2.7 in the future, would our code be backward compatible?\nSince OpenLineage has now moved inside airflow and I think there is no concept of extractors in the latest version.", + "user": "U05A1D80QKF", + "ts": "1695276670.439269", + "blocks": [ + { + "type": "rich_text", + "block_id": "bqXzF", + "elements": [ + { + "type": "rich_text_section", + "elements": [ + { + "type": "text", + "text": "Hi, we're using Custom Operators in airflow(2.5) and are planning to expose lineage via default extractors: " + }, + { + "type": "link", + "url": "https://openlineage.io/docs/integrations/airflow/default-extractors/" + }, + { + "type": "text", + "text": "\n" + }, + { + "type": "text", + "text": "Question", + "style": { + "bold": true + } + }, + { + "type": "text", + "text": ": Now if we upgrade our Airflow version to 2.7 in the future, would our code be backward compatible?\nSince OpenLineage has now moved inside airflow and I think there is no concept of extractors in the latest version." + } + ] + } + ] + } + ], + "team": "T01CWUYP5AR", + "thread_ts": "1695276670.439269", + "reply_count": 2, + "reply_users_count": 2, + "latest_reply": "1695823449.348469", + "reply_users": [ + "U05A1D80QKF", + "U02S6F54MAB" + ], + "is_locked": false, + "subscribed": false, + "replies": [ + { + "client_msg_id": "d85590cd-9f1f-44e7-a019-192a58b617a8", + "type": "message", + "text": "Also, do we have any docs on how OL works with the latest airflow version? Few questions:\n• How is it replacing the concept of custom extractors and Manually Annotated Lineage in the latest version? \n• Do we have any examples of setting up the integration to emit input/output datasets for non supported Operators like PythonOperator?", + "user": "U05A1D80QKF", + "ts": "1695276900.832599", + "blocks": [ + { + "type": "rich_text", + "block_id": "up+gD", + "elements": [ + { + "type": "rich_text_section", + "elements": [ + { + "type": "text", + "text": "Also, do we have any docs on how OL works with the latest airflow version? Few questions:\n" + } + ] + }, + { + "type": "rich_text_list", + "elements": [ + { + "type": "rich_text_section", + "elements": [ + { + "type": "text", + "text": "How is it replacing the concept of custom extractors and Manually Annotated Lineage in the latest version? " + } + ] + }, + { + "type": "rich_text_section", + "elements": [ + { + "type": "text", + "text": "Do we have any examples of setting up the integration to emit input/output datasets for non supported Operators like PythonOperator?" + } + ] + } + ], + "style": "bullet", + "indent": 0, + "border": 0 + } + ] + } + ], + "team": "T01CWUYP5AR", + "thread_ts": "1695276670.439269", + "parent_user_id": "U05A1D80QKF" + }, + { + "client_msg_id": "d47f79ef-74de-45fa-b03c-49eef46ecf40", + "type": "message", + "text": "> Question: Now if we upgrade our Airflow version to 2.7 in the future, would our code be backward compatible?\nIt will be compatible, “default extractors” is generally the same concept as we’re using in the 2.7 integration.\nOne thing that might be good to update is import paths, from `openlineage.airflow` to `airflow.providers.openlineage` but should work both ways\n\n> • Do we have any code samples/docs of setting up the integration to emit input/output datasets for non supported Operators like PythonOperator?\nOur experience with that is currently lacking - this means, it works like in bare airflow, if you annotate your PythonOperator tasks with old Airflow lineage like \n\nWe want to make this experience better - by doing few things\n• instrumenting hooks, then collecting lineage from them\n• integration with AIP-48 datasets\n• allowing to emit lineage collected inside Airflow task by other means, by providing core Airflow API for that\nAll those things require changing core Airflow in a couple of ways:\n• tracking which hooks were used during PythonOperator execution\n• just being able to emit datasets (airflow inlets/outlets) from inside of a task - they are now a static thing, so if you try that it does not work\n• providing better API for emitting that lineage, preferably based on OpenLineage itself rather than us having to convert that later.\nAs this requires core Airflow changes, it won’t be live until Airflow 2.8 at the earliest.\n\nthanks to <@U01RA9B5GG2> for this response", + "user": "U02S6F54MAB", + "ts": "1695823449.348469", + "blocks": [ + { + "type": "rich_text", + "block_id": "1nu2w", + "elements": [ + { + "type": "rich_text_quote", + "elements": [ + { + "type": "text", + "text": "Question: Now if we upgrade our Airflow version to 2.7 in the future, would our code be backward compatible?" + } + ] + }, + { + "type": "rich_text_section", + "elements": [ + { + "type": "text", + "text": "It will be compatible, “default extractors” is generally the same concept as we’re using in the 2.7 integration.\nOne thing that might be good to update is import paths, from " + }, + { + "type": "text", + "text": "openlineage.airflow", + "style": { + "code": true + } + }, + { + "type": "text", + "text": " to " + }, + { + "type": "text", + "text": "airflow.providers.openlineage", + "style": { + "code": true + } + }, + { + "type": "text", + "text": " but should work both ways\n\n" + } + ] + }, + { + "type": "rich_text_list", + "elements": [ + { + "type": "rich_text_section", + "elements": [ + { + "type": "text", + "text": "Do we have any code samples/docs of setting up the integration to emit input/output datasets for non supported Operators like PythonOperator?" + } + ] + } + ], + "style": "bullet", + "indent": 0, + "border": 1 + }, + { + "type": "rich_text_section", + "elements": [ + { + "type": "text", + "text": "Our experience with that is currently lacking - this means, it works like in bare airflow, if you annotate your PythonOperator tasks with old Airflow lineage like " + }, + { + "type": "link", + "url": "https://airflow.apache.org/docs/apache-airflow/stable/administration-and-deployment/lineage.html", + "text": "in this doc." + }, + { + "type": "text", + "text": "\n\nWe want to make this experience better - by doing few things\n" + } + ] + }, + { + "type": "rich_text_list", + "elements": [ + { + "type": "rich_text_section", + "elements": [ + { + "type": "text", + "text": "instrumenting hooks, then collecting lineage from them" + } + ] + }, + { + "type": "rich_text_section", + "elements": [ + { + "type": "text", + "text": "integration with AIP-48 datasets" + } + ] + }, + { + "type": "rich_text_section", + "elements": [ + { + "type": "text", + "text": "allowing to emit lineage collected inside Airflow task by other means, by providing core Airflow API for that" + } + ] + } + ], + "style": "bullet", + "indent": 0, + "border": 0 + }, + { + "type": "rich_text_section", + "elements": [ + { + "type": "text", + "text": "\nAll those things require changing core Airflow in a couple of ways:\n" + } + ] + }, + { + "type": "rich_text_list", + "elements": [ + { + "type": "rich_text_section", + "elements": [ + { + "type": "text", + "text": "tracking which hooks were used during PythonOperator execution" + } + ] + }, + { + "type": "rich_text_section", + "elements": [ + { + "type": "text", + "text": "just being able to emit datasets (airflow inlets/outlets) from inside of a task - they are now a static thing, so if you try that it does not work" + } + ] + }, + { + "type": "rich_text_section", + "elements": [ + { + "type": "text", + "text": "providing better API for emitting that lineage, preferably based on OpenLineage itself rather than us having to convert that later." + } + ] + } + ], + "style": "bullet", + "indent": 0, + "border": 0 + }, + { + "type": "rich_text_section", + "elements": [ + { + "type": "text", + "text": "\nAs this requires core Airflow changes, it won’t be live until Airflow 2.8 at the earliest.\n\nthanks to " + }, + { + "type": "user", + "user_id": "U01RA9B5GG2" + }, + { + "type": "text", + "text": " for this response" + } + ] + } + ] + } + ], + "team": "T01CWUYP5AR", + "thread_ts": "1695276670.439269", + "parent_user_id": "U05A1D80QKF" + } + ] + }, + { + "client_msg_id": "f6b7340b-e7c9-4126-8001-1ac903fc62bc", + "type": "message", + "text": "\nWe released OpenLineage 1.2.2!\nAdded\n• Spark: publish the `ProcessingEngineRunFacet` as part of the normal operation of the `OpenLineageSparkEventListener` `#2089` \n• Spark: capture and emit `spark.databricks.clusterUsageTags.clusterAllTags` variable from databricks environment `#2099` \nFixed\n• Common: support parsing dbt_project.yml without target-path `#2106` \n• Proxy: fix Proxy chart `#2091` \n• Python: fix serde filtering `#2044` \n• Python: use non-deprecated `apiKey` if loading it from env variables `@2029` \n• Spark: Improve RDDs on S3 integration. `#2039` \n• Flink: prevent sending `running` events after job completes `#2075` \n• Spark & Flink: Unify dataset naming from URI objects `#2083` \n• Spark: Databricks improvements `#2076` \nRemoved\n• SQL: remove sqlparser dependency from iface-java and iface-py `#2090` \nThanks to all the contributors, including new contributors <@U055N2GRT4P>, , and !\n*Release:* \n*Changelog:* \n*Commit history:* \n*Maven:* \n*PyPI:* ", + "user": "U02LXF3HUN7", + "ts": "1695244138.650089", + "blocks": [ + { + "type": "rich_text", + "block_id": "BzgIj", + "elements": [ + { + "type": "rich_text_section", + "elements": [ + { + "type": "broadcast", + "range": "channel" + }, + { + "type": "text", + "text": "\nWe released OpenLineage 1.2.2!\nAdded\n" + } + ] + }, + { + "type": "rich_text_list", + "elements": [ + { + "type": "rich_text_section", + "elements": [ + { + "type": "text", + "text": "Spark: publish the " + }, + { + "type": "text", + "text": "ProcessingEngineRunFacet", + "style": { + "code": true + } + }, + { + "type": "text", + "text": " as part of the normal operation of the " + }, + { + "type": "text", + "text": "OpenLineageSparkEventListener", + "style": { + "code": true + } + }, + { + "type": "text", + "text": " " + }, + { + "type": "text", + "text": "#2089", + "style": { + "code": true + } + }, + { + "type": "text", + "text": " " + }, + { + "type": "link", + "url": "https://github.com/d-m-h", + "text": "@d-m-h", + "unsafe": true + } + ] + }, + { + "type": "rich_text_section", + "elements": [ + { + "type": "text", + "text": "Spark: capture and emit " + }, + { + "type": "text", + "text": "spark.databricks.clusterUsageTags.clusterAllTags", + "style": { + "code": true + } + }, + { + "type": "text", + "text": " variable from databricks environment " + }, + { + "type": "text", + "text": "#2099", + "style": { + "code": true + } + }, + { + "type": "text", + "text": " " + }, + { + "type": "link", + "url": "https://github.com/Anirudh181001", + "text": "@Anirudh181001", + "unsafe": true + } + ] + } + ], + "style": "bullet", + "indent": 0, + "border": 0 + }, + { + "type": "rich_text_section", + "elements": [ + { + "type": "text", + "text": "Fixed\n" + } + ] + }, + { + "type": "rich_text_list", + "elements": [ + { + "type": "rich_text_section", + "elements": [ + { + "type": "text", + "text": "Common: support parsing dbt_project.yml without target-path " + }, + { + "type": "text", + "text": "#2106", + "style": { + "code": true + } + }, + { + "type": "text", + "text": " " + }, + { + "type": "link", + "url": "https://github.com/tatiana", + "text": "@tatiana", + "unsafe": true + } + ] + }, + { + "type": "rich_text_section", + "elements": [ + { + "type": "text", + "text": "Proxy: fix Proxy chart " + }, + { + "type": "text", + "text": "#2091", + "style": { + "code": true + } + }, + { + "type": "text", + "text": " " + }, + { + "type": "link", + "url": "https://github.com/harels", + "text": "@harels", + "unsafe": true + } + ] + }, + { + "type": "rich_text_section", + "elements": [ + { + "type": "text", + "text": "Python: fix serde filtering " + }, + { + "type": "text", + "text": "#2044", + "style": { + "code": true + } + }, + { + "type": "text", + "text": " " + }, + { + "type": "link", + "url": "https://github.com/xli-1026", + "text": "@xli-1026", + "unsafe": true + } + ] + }, + { + "type": "rich_text_section", + "elements": [ + { + "type": "text", + "text": "Python: use non-deprecated " + }, + { + "type": "text", + "text": "apiKey", + "style": { + "code": true + } + }, + { + "type": "text", + "text": " if loading it from env variables " + }, + { + "type": "text", + "text": "@2029", + "style": { + "code": true + } + }, + { + "type": "text", + "text": " " + }, + { + "type": "link", + "url": "https://github.com/mobuchowski", + "text": "@mobuchowski", + "unsafe": true + } + ] + }, + { + "type": "rich_text_section", + "elements": [ + { + "type": "text", + "text": "Spark: Improve RDDs on S3 integration. " + }, + { + "type": "text", + "text": "#2039", + "style": { + "code": true + } + }, + { + "type": "text", + "text": " " + }, + { + "type": "link", + "url": "https://github.com/pawel-big-lebowski", + "text": "@pawel-big-lebowski", + "unsafe": true + } + ] + }, + { + "type": "rich_text_section", + "elements": [ + { + "type": "text", + "text": "Flink: prevent sending " + }, + { + "type": "text", + "text": "running", + "style": { + "code": true + } + }, + { + "type": "text", + "text": " events after job completes " + }, + { + "type": "text", + "text": "#2075", + "style": { + "code": true + } + }, + { + "type": "text", + "text": " " + }, + { + "type": "link", + "url": "https://github.com/pawel-big-lebowski", + "text": "@pawel-big-lebowski", + "unsafe": true + } + ] + }, + { + "type": "rich_text_section", + "elements": [ + { + "type": "text", + "text": "Spark & Flink: Unify dataset naming from URI objects " + }, + { + "type": "text", + "text": "#2083", + "style": { + "code": true + } + }, + { + "type": "text", + "text": " " + }, + { + "type": "link", + "url": "https://github.com/pawel-big-lebowski", + "text": "@pawel-big-lebowski", + "unsafe": true + } + ] + }, + { + "type": "rich_text_section", + "elements": [ + { + "type": "text", + "text": "Spark: Databricks improvements " + }, + { + "type": "text", + "text": "#2076", + "style": { + "code": true + } + }, + { + "type": "text", + "text": " " + }, + { + "type": "link", + "url": "https://github.com/pawel-big-lebowski", + "text": "@pawel-big-lebowski", + "unsafe": true + } + ] + } + ], + "style": "bullet", + "indent": 0, + "border": 0 + }, + { + "type": "rich_text_section", + "elements": [ + { + "type": "text", + "text": "Removed\n" + } + ] + }, + { + "type": "rich_text_list", + "elements": [ + { + "type": "rich_text_section", + "elements": [ + { + "type": "text", + "text": "SQL: remove sqlparser dependency from iface-java and iface-py " + }, + { + "type": "text", + "text": "#2090", + "style": { + "code": true + } + }, + { + "type": "text", + "text": " " + }, + { + "type": "link", + "url": "https://github.com/JDarDagran", + "text": "@JDarDagran", + "unsafe": true + } + ] + } + ], + "style": "bullet", + "indent": 0, + "border": 0 + }, + { + "type": "rich_text_section", + "elements": [ + { + "type": "text", + "text": "Thanks to all the contributors, including new contributors " + }, + { + "type": "user", + "user_id": "U055N2GRT4P" + }, + { + "type": "text", + "text": ", " + }, + { + "type": "link", + "url": "https://github.com/xli-1026", + "text": "@xli-1026", + "unsafe": true + }, + { + "type": "text", + "text": ", and " + }, + { + "type": "link", + "url": "https://github.com/d-m-h", + "text": "@d-m-h", + "unsafe": true + }, + { + "type": "text", + "text": "!\n" + }, + { + "type": "text", + "text": "Release:", + "style": { + "bold": true + } + }, + { + "type": "text", + "text": " " + }, + { + "type": "link", + "url": "https://github.com/OpenLineage/OpenLineage/releases/tag/1.2.2" + }, + { + "type": "text", + "text": "\n" + }, + { + "type": "text", + "text": "Changelog: ", + "style": { + "bold": true + } + }, + { + "type": "link", + "url": "https://github.com/OpenLineage/OpenLineage/blob/main/CHANGELOG.md" + }, + { + "type": "text", + "text": "\n" + }, + { + "type": "text", + "text": "Commit history:", + "style": { + "bold": true + } + }, + { + "type": "text", + "text": " " + }, + { + "type": "link", + "url": "https://github.com/OpenLineage/OpenLineage/compare/1.1.0...1.2.0", + "text": "https://github.com/OpenLineage/OpenLineage/compare/1.1.0...1.2.2" + }, + { + "type": "text", + "text": "\n" + }, + { + "type": "text", + "text": "Maven:", + "style": { + "bold": true + } + }, + { + "type": "text", + "text": " " + }, + { + "type": "link", + "url": "https://oss.sonatype.org/#nexus-search;quick~openlineage" + }, + { + "type": "text", + "text": "\n" + }, + { + "type": "text", + "text": "PyPI:", + "style": { + "bold": true + } + }, + { + "type": "text", + "text": " " + }, + { + "type": "link", + "url": "https://pypi.org/project/openlineage-python/" + } + ] + } + ] + } + ], + "team": "T01CWUYP5AR", + "thread_ts": "1695244138.650089", + "reply_count": 3, + "reply_users_count": 3, + "latest_reply": "1695471077.136259", + "reply_users": [ + "U05ST398BHT", + "U02LXF3HUN7", + "U01RA9B5GG2" + ], + "is_locked": false, + "subscribed": true, + "last_read": "1695471077.136259", + "reactions": [ + { + "name": "fire", + "users": [ + "U01RA9B5GG2", + "U01HNKK4XAM", + "U04EZ2LPDV4" + ], + "count": 3 + }, + { + "name": "+1", + "users": [ + "U05QL7LN2GH", + "U05GY00DY6A", + "U05SMTVPPL3" + ], + "count": 3 + } + ], + "replies": [ + { + "client_msg_id": "3472acf2-f11c-4e4c-8d6b-24ce1c40d93f", + "type": "message", + "text": "Hi <@U02LXF3HUN7> Thank you! I love the job that you’ve done. If you have a few seconds, please hint at how I can push lineage gathered from Airflow and Spark jobs into DataHub for visualization? I didn’t find any solutions or official support neither at Openlineage nor at DataHub, but I still want to continue using Openlineage", + "user": "U05ST398BHT", + "ts": "1695431120.413409", + "blocks": [ + { + "type": "rich_text", + "block_id": "eZxY8", + "elements": [ + { + "type": "rich_text_section", + "elements": [ + { + "type": "text", + "text": "Hi " + }, + { + "type": "user", + "user_id": "U02LXF3HUN7" + }, + { + "type": "text", + "text": " Thank you! I love the job that you’ve done. If you have a few seconds, please hint at how I can push lineage gathered from Airflow and Spark jobs into DataHub for visualization? I didn’t find any solutions or official support neither at Openlineage nor at DataHub, but I still want to continue using Openlineage" + } + ] + } + ] + } + ], + "team": "T01CWUYP5AR", + "thread_ts": "1695244138.650089", + "parent_user_id": "U02LXF3HUN7" + }, + { + "client_msg_id": "7fc32230-ea04-4080-9cd1-7a4c616bed49", + "type": "message", + "text": "Hi Yevhenii, thank you for using OpenLineage. The DataHub integration is new to us, but perhaps the experts on Spark and Airflow know more. <@U02MK6YNAQ5> <@U01RA9B5GG2> <@U02S6F54MAB>", + "user": "U02LXF3HUN7", + "ts": "1695432622.381059", + "blocks": [ + { + "type": "rich_text", + "block_id": "M5Py7", + "elements": [ + { + "type": "rich_text_section", + "elements": [ + { + "type": "text", + "text": "Hi Yevhenii, thank you for using OpenLineage. The DataHub integration is new to us, but perhaps the experts on Spark and Airflow know more. " + }, + { + "type": "user", + "user_id": "U02MK6YNAQ5" + }, + { + "type": "text", + "text": " " + }, + { + "type": "user", + "user_id": "U01RA9B5GG2" + }, + { + "type": "text", + "text": " " + }, + { + "type": "user", + "user_id": "U02S6F54MAB" + } + ] + } + ] + } + ], + "team": "T01CWUYP5AR", + "edited": { + "user": "U02LXF3HUN7", + "ts": "1695489116.000000" + }, + "thread_ts": "1695244138.650089", + "parent_user_id": "U02LXF3HUN7" + }, + { + "client_msg_id": "2b9073be-55e3-41dc-8200-4182e30164ab", + "type": "message", + "text": "<@U05ST398BHT> at Airflow Summit, Shirshanka Das from DataHub mentioned this as upcoming feature.", + "user": "U01RA9B5GG2", + "ts": "1695471077.136259", + "blocks": [ + { + "type": "rich_text", + "block_id": "EdLPn", + "elements": [ + { + "type": "rich_text_section", + "elements": [ + { + "type": "user", + "user_id": "U05ST398BHT" + }, + { + "type": "text", + "text": " at Airflow Summit, Shirshanka Das from DataHub mentioned this as upcoming feature." + } + ] + } + ] + } + ], + "team": "T01CWUYP5AR", + "thread_ts": "1695244138.650089", + "parent_user_id": "U02LXF3HUN7", + "reactions": [ + { + "name": "+1", + "users": [ + "U05ST398BHT" + ], + "count": 1 + }, + { + "name": "dart", + "users": [ + "U05ST398BHT" + ], + "count": 1 + } + ] + } + ] + }, + { + "client_msg_id": "6bc2e8a7-85c8-45cd-808d-351adc7188de", + "type": "message", + "text": "congrats folks :partying_face: ", + "user": "U05HFGKEYVB", + "ts": "1695217014.549799", + "blocks": [ + { + "type": "rich_text", + "block_id": "D8SPh", + "elements": [ + { + "type": "rich_text_section", + "elements": [ + { + "type": "text", + "text": "congrats folks " + }, + { + "type": "emoji", + "name": "partying_face", + "unicode": "1f973" + }, + { + "type": "text", + "text": " " + }, + { + "type": "link", + "url": "https://lfaidata.foundation/blog/2023/09/20/lf-ai-data-foundation-announces-graduation-of-openlineage-project" + } + ] + } + ] + } + ], + "team": "T01CWUYP5AR", + "reactions": [ + { + "name": "tada", + "users": [ + "U02S6F54MAB", + "U01HVNU6A4C", + "U053LLVTHRN", + "U05QL7LN2GH", + "U01SW738WCF", + "U01RA9B5GG2", + "U05JBHLPY8K", + "U04AZ7992SU", + "U01HNKK4XAM", + "U02K353H2KF", + "U01DPTNCGU8" + ], + "count": 11 + }, + { + "name": "+1", + "users": [ + "U05JBHLPY8K" + ], + "count": 1 + }, + { + "name": "heart", + "users": [ + "U01HNKK4XAM" + ], + "count": 1 + } + ] + }, + { + "client_msg_id": "18d20727-c8c6-4490-aea2-b1db03cd54c8", + "type": "message", + "text": "Hi I need help in extracting OpenLineage for PostgresOperator in json format.\nany suggestions or comments would be greatly appreciated", + "user": "U05SQGH8DV4", + "ts": "1695106067.665469", + "blocks": [ + { + "type": "rich_text", + "block_id": "+DToP", + "elements": [ + { + "type": "rich_text_section", + "elements": [ + { + "type": "text", + "text": "Hi I need help in extracting OpenLineage for PostgresOperator in json format.\nany suggestions or comments would be greatly appreciated" + } + ] + } + ] + } + ], + "team": "T01CWUYP5AR", + "thread_ts": "1695106067.665469", + "reply_count": 3, + "reply_users_count": 2, + "latest_reply": "1695205616.828789", + "reply_users": [ + "U01RA9B5GG2", + "U05SQGH8DV4" + ], + "is_locked": false, + "subscribed": false, + "replies": [ + { + "client_msg_id": "cbb8f343-a8b8-4734-85d6-a53beb93c9f7", + "type": "message", + "text": "If you're using Airflow 2.7, take a look at ", + "user": "U01RA9B5GG2", + "ts": "1695156006.370219", + "blocks": [ + { + "type": "rich_text", + "block_id": "7mV17", + "elements": [ + { + "type": "rich_text_section", + "elements": [ + { + "type": "text", + "text": "If you're using Airflow 2.7, take a look at " + }, + { + "type": "link", + "url": "https://airflow.apache.org/docs/apache-airflow-providers-openlineage/stable/guides/user.html", + "text": "https://airflow.apache.org/docs/apache-airflow-providers-openlineage/stable/guides/user.html" + } + ] + } + ] + } + ], + "team": "T01CWUYP5AR", + "thread_ts": "1695106067.665469", + "parent_user_id": "U05SQGH8DV4", + "reactions": [ + { + "name": "heart", + "users": [ + "U05SQGH8DV4" + ], + "count": 1 + } + ] + }, + { + "client_msg_id": "accd82a6-a042-4f83-af6a-05efe3b0b460", + "type": "message", + "text": "If you use one of the lower versions, take a look here ", + "user": "U01RA9B5GG2", + "ts": "1695156054.864799", + "blocks": [ + { + "type": "rich_text", + "block_id": "HT53+", + "elements": [ + { + "type": "rich_text_section", + "elements": [ + { + "type": "text", + "text": "If you use one of the lower versions, take a look here " + }, + { + "type": "link", + "url": "https://openlineage.io/docs/integrations/airflow/usage", + "text": "https://openlineage.io/docs/integrations/airflow/usage" + } + ] + } + ] + } + ], + "team": "T01CWUYP5AR", + "attachments": [ + { + "from_url": "https://openlineage.io/docs/integrations/airflow/usage", + "service_icon": "https://openlineage.io/img/favicon.ico", + "id": 1, + "original_url": "https://openlineage.io/docs/integrations/airflow/usage", + "fallback": "Using the Airflow integration | OpenLineage", + "text": "PREREQUISITES", + "title": "Using the Airflow integration | OpenLineage", + "title_link": "https://openlineage.io/docs/integrations/airflow/usage", + "service_name": "openlineage.io" + } + ], + "thread_ts": "1695106067.665469", + "parent_user_id": "U05SQGH8DV4" + }, + { + "client_msg_id": "24180d5a-cca8-4d65-a29e-300ff58b6059", + "type": "message", + "text": "Maciej,\nThanks for sharing the link \nthis should address the issue", + "user": "U05SQGH8DV4", + "ts": "1695205616.828789", + "blocks": [ + { + "type": "rich_text", + "block_id": "EWnYW", + "elements": [ + { + "type": "rich_text_section", + "elements": [ + { + "type": "text", + "text": "Maciej,\nThanks for sharing the link " + }, + { + "type": "link", + "url": "https://airflow.apache.org/docs/apache-airflow-providers-openlineage/stable/guides/user.html" + }, + { + "type": "text", + "text": "\nthis should address the issue" + } + ] + } + ] + } + ], + "team": "T01CWUYP5AR", + "thread_ts": "1695106067.665469", + "parent_user_id": "U05SQGH8DV4" + } + ] + }, + { + "client_msg_id": "a02784a5-f80b-4209-bc92-dd5ae76b87d9", + "type": "message", + "text": "Hi! I'm in need of help with wrapping my head around OpenLineage. My team have the goal of collecting metadata from the Airflow operators GreatExpectationsOperator, PythonOperator, MsSqlOperator and BashOperator (for dbt). Where can I see the sourcecode for what is collected for each operator, and is there support for these in the new provider *apache-airflow-providers-openlineage*? I am super confused and feel lost in the docs. :exploding_head: We are using MSSQL/ODBC to connect to our db, and this data does not seem to appear as datasets in Marquez, do I need to configure this? If so, HOW and WHERE? :smiling_face_with_tear:\n\nHappy for any help, big or small! :pray:", + "user": "U05K8F1T887", + "ts": "1695039754.591479", + "blocks": [ + { + "type": "rich_text", + "block_id": "iTsci", + "elements": [ + { + "type": "rich_text_section", + "elements": [ + { + "type": "text", + "text": "Hi! I'm in need of help with wrapping my head around OpenLineage. My team have the goal of collecting metadata from the Airflow operators GreatExpectationsOperator, PythonOperator, MsSqlOperator and BashOperator (for dbt). Where can I see the sourcecode for what is collected for each operator, and is there support for these in the new provider" + }, + { + "type": "text", + "text": " apache-airflow-providers-openlineage", + "style": { + "bold": true + } + }, + { + "type": "text", + "text": "? I am super confused and feel lost in the docs. " + }, + { + "type": "emoji", + "name": "exploding_head", + "unicode": "1f92f" + }, + { + "type": "text", + "text": " We are using MSSQL/ODBC to connect to our db, and this data does not seem to appear as datasets in Marquez, do I need to configure this? If so, HOW and WHERE? " + }, + { + "type": "emoji", + "name": "smiling_face_with_tear", + "unicode": "1f972" + }, + { + "type": "text", + "text": "\n\nHappy for any help, big or small! " + }, + { + "type": "emoji", + "name": "pray", + "unicode": "1f64f" + } + ] + } + ] + } + ], + "team": "T01CWUYP5AR", + "thread_ts": "1695039754.591479", + "reply_count": 3, + "reply_users_count": 1, + "latest_reply": "1695068818.187449", + "reply_users": [ + "U02S6F54MAB" + ], + "is_locked": false, + "subscribed": false, + "replies": [ + { + "client_msg_id": "985ce8ec-0abb-4dcb-a0bc-eb897bf403a2", + "type": "message", + "text": "there’s no actual single source of what integrations are currently implemented in openlineage Airflow provider. That’s something we should work on so it’s more visible", + "user": "U02S6F54MAB", + "ts": "1695068767.883209", + "blocks": [ + { + "type": "rich_text", + "block_id": "c5rkV", + "elements": [ + { + "type": "rich_text_section", + "elements": [ + { + "type": "text", + "text": "there’s no actual single source of what integrations are currently implemented in openlineage Airflow provider. That’s something we should work on so it’s more visible" + } + ] + } + ] + } + ], + "team": "T01CWUYP5AR", + "thread_ts": "1695039754.591479", + "parent_user_id": "U05K8F1T887" + }, + { + "client_msg_id": "c5d88209-70c9-4f31-b4fd-eed402f485d3", + "type": "message", + "text": "answering this quickly - GE & MS SQL are not currently implemented yet in the provider", + "user": "U02S6F54MAB", + "ts": "1695068806.046349", + "blocks": [ + { + "type": "rich_text", + "block_id": "3pyGD", + "elements": [ + { + "type": "rich_text_section", + "elements": [ + { + "type": "text", + "text": "answering this quickly - GE & MS SQL are not currently implemented yet in the provider" + } + ] + } + ] + } + ], + "team": "T01CWUYP5AR", + "thread_ts": "1695039754.591479", + "parent_user_id": "U05K8F1T887" + }, + { + "client_msg_id": "9acf490a-9219-4325-a53c-8b68b69c4200", + "type": "message", + "text": "but I also invite you to contribute if you’re interested! :slightly_smiling_face:", + "user": "U02S6F54MAB", + "ts": "1695068818.187449", + "blocks": [ + { + "type": "rich_text", + "block_id": "NkKHD", + "elements": [ + { + "type": "rich_text_section", + "elements": [ + { + "type": "text", + "text": "but I also invite you to contribute if you’re interested! " + }, + { + "type": "emoji", + "name": "slightly_smiling_face", + "unicode": "1f642" + } + ] + } + ] + } + ], + "team": "T01CWUYP5AR", + "edited": { + "user": "U02S6F54MAB", + "ts": "1695068825.000000" + }, + "thread_ts": "1695039754.591479", + "parent_user_id": "U05K8F1T887" + } + ] + }, + { + "client_msg_id": "00f92226-f107-42e8-bb34-7adddf13bc0b", + "type": "message", + "text": "It doesn't seem like there's a way to override the OL endpoint from the default (`/api/v1/lineage`) in Airflow? I tried setting the `OPENLINEAGE_ENDPOINT` environment to no avail. Based on , it seems that only `OPENLINEAGE_URL` was used to construct `HttpConfig` ?", + "user": "U01HVNU6A4C", + "ts": "1694956061.909169", + "blocks": [ + { + "type": "rich_text", + "block_id": "1FrxC", + "elements": [ + { + "type": "rich_text_section", + "elements": [ + { + "type": "text", + "text": "It doesn't seem like there's a way to override the OL endpoint from the default (" + }, + { + "type": "text", + "text": "/api/v1/lineage", + "style": { + "code": true + } + }, + { + "type": "text", + "text": ") in Airflow? I tried setting the " + }, + { + "type": "text", + "text": "OPENLINEAGE_ENDPOINT", + "style": { + "code": true + } + }, + { + "type": "text", + "text": " environment to no avail. Based on " + }, + { + "type": "link", + "url": "https://github.com/OpenLineage/OpenLineage/blob/main/client/python/openlineage/client/transport/factory.py#L80-L87", + "text": "this statement" + }, + { + "type": "text", + "text": ", it seems that only " + }, + { + "type": "text", + "text": "OPENLINEAGE_URL", + "style": { + "code": true + } + }, + { + "type": "text", + "text": " was used to construct " + }, + { + "type": "text", + "text": "HttpConfig", + "style": { + "code": true + } + }, + { + "type": "text", + "text": " ?" + } + ] + } + ] + } + ], + "team": "T01CWUYP5AR", + "attachments": [ + { + "id": 1, + "footer_icon": "https://slack.github.com/static/img/favicon-neutral.png", + "color": "24292f", + "bot_id": "B01VA0FB340", + "app_unfurl_url": "https://github.com/OpenLineage/OpenLineage/blob/main/client/python/openlineage/client/transport/factory.py#L80-L87", + "is_app_unfurl": true, + "app_id": "A01BP7R4KNY", + "fallback": "", + "text": "```\n config = HttpConfig(\n url=os.environ[\"OPENLINEAGE_URL\"],\n auth=create_token_provider(\n {\n \"type\": \"api_key\",\n \"apiKey\": os.environ.get(\"OPENLINEAGE_API_KEY\", \"\"),\n },\n ),\n```", + "title": "", + "footer": "", + "mrkdwn_in": [ + "text" + ] + } + ], + "thread_ts": "1694956061.909169", + "reply_count": 4, + "reply_users_count": 2, + "latest_reply": "1696337536.498149", + "reply_users": [ + "U02S6F54MAB", + "U01HVNU6A4C" + ], + "is_locked": false, + "subscribed": false, + "replies": [ + { + "client_msg_id": "2a1c7601-9128-4280-a927-65d6f9f23578", + "type": "message", + "text": "That’s correct. For now there’s no way to configure the endpoint via env var. You can do that by using config file", + "user": "U02S6F54MAB", + "ts": "1695068711.558639", + "blocks": [ + { + "type": "rich_text", + "block_id": "nsSnE", + "elements": [ + { + "type": "rich_text_section", + "elements": [ + { + "type": "text", + "text": "That’s correct. For now there’s no way to configure the endpoint via env var. You can do that by using config file" + } + ] + } + ] + } + ], + "team": "T01CWUYP5AR", + "thread_ts": "1694956061.909169", + "parent_user_id": "U01HVNU6A4C" + }, + { + "client_msg_id": "a0a9275d-9cf8-4582-acda-0b8c7b7d058b", + "type": "message", + "text": "How do you do that in Airflow? Any particular reason for excluding endpoint override via env var? Happy to create a PR to fix that.", + "user": "U01HVNU6A4C", + "ts": "1695069039.150789", + "blocks": [ + { + "type": "rich_text", + "block_id": "gjGUd", + "elements": [ + { + "type": "rich_text_section", + "elements": [ + { + "type": "text", + "text": "How do you do that in Airflow? Any particular reason for excluding endpoint override via env var? Happy to create a PR to fix that." + } + ] + } + ] + } + ], + "team": "T01CWUYP5AR", + "thread_ts": "1694956061.909169", + "parent_user_id": "U01HVNU6A4C" + }, + { + "client_msg_id": "8113b26a-ebec-4395-86d6-697483fa2e29", + "type": "message", + "text": "historical I guess? go for the PR, of course :rocket:", + "user": "U02S6F54MAB", + "ts": "1695070368.703459", + "blocks": [ + { + "type": "rich_text", + "block_id": "Vnr4T", + "elements": [ + { + "type": "rich_text_section", + "elements": [ + { + "type": "text", + "text": "historical I guess? go for the PR, of course " + }, + { + "type": "emoji", + "name": "rocket", + "unicode": "1f680" + } + ] + } + ] + } + ], + "team": "T01CWUYP5AR", + "thread_ts": "1694956061.909169", + "parent_user_id": "U01HVNU6A4C" + }, + { + "client_msg_id": "9b181669-3aad-4e22-8202-3bc76c3567cd", + "type": "message", + "text": "", + "user": "U01HVNU6A4C", + "ts": "1696337536.498149", + "blocks": [ + { + "type": "rich_text", + "block_id": "iw279", + "elements": [ + { + "type": "rich_text_section", + "elements": [ + { + "type": "link", + "url": "https://github.com/OpenLineage/OpenLineage/pull/2151" + } + ] + } + ] + } + ], + "team": "T01CWUYP5AR", + "attachments": [ + { + "id": 1, + "footer_icon": "https://slack.github.com/static/img/favicon-neutral.png", + "ts": 1696337382, + "color": "36a64f", + "bot_id": "B01VA0FB340", + "app_unfurl_url": "https://github.com/OpenLineage/OpenLineage/pull/2151", + "is_app_unfurl": true, + "app_id": "A01BP7R4KNY", + "fallback": "#2151 Allow setting client's endpoint via environment variable", + "text": "*Problem*\n\nCurrently, it's not possible to set the OpenLineage endpoint (hard-coded to `/api/v1/lineage`) using an environment variable when running the Airflow integration.\n\n*Solution*\n\nGiven that it's not possible to create the client manually in Airflow, especially now that OpenLineage has become an official Airflow provider, this change seems like the only feasible solution.\n\n☐ Your change modifies the OpenLineage model\n☐ Your change modifies one or more OpenLineage \n\n*One-line summary:*\n\nAllow setting client's endpoint via environment variable.\n\n*Checklist*\n\n☑︎ You've your work\n☑︎ Your pull request title follows our \n☑︎ Your changes are accompanied by tests (_if relevant_)\n☑︎ Your change contains a and is self-contained\n☑︎ You've updated any relevant documentation (_if relevant_)\n☑︎ Your comment includes a one-liner for the changelog about the specific purpose of the change (_if necessary_)\n☐ You've versioned the core OpenLineage model or facets according to (_if relevant_)\n☐ You've added a to source files (_if relevant_)\n\n* * *\n\nSPDX-License-Identifier: Apache-2.0 \nCopyright 2018-2023 contributors to the OpenLineage project", + "title": "#2151 Allow setting client's endpoint via environment variable", + "title_link": "https://github.com/OpenLineage/OpenLineage/pull/2151", + "footer": "", + "fields": [ + { + "value": "documentation, client/python", + "title": "Labels", + "short": true + } + ], + "mrkdwn_in": [ + "text" + ] + } + ], + "thread_ts": "1694956061.909169", + "parent_user_id": "U01HVNU6A4C" + } + ] + }, + { + "client_msg_id": "2288c4d1-0b22-4a10-90fb-914438bbfc50", + "type": "message", + "text": " is there a way by which we could add custom headers to openlineage client in airflow, i see that provision is there for spark integration via these properties --> abcdef", + "user": "U05QL7LN2GH", + "ts": "1694907627.974239", + "blocks": [ + { + "type": "rich_text", + "block_id": "ZvNmS", + "elements": [ + { + "type": "rich_text_section", + "elements": [ + { + "type": "broadcast", + "range": "here" + }, + { + "type": "text", + "text": " is there a way by which we could add custom headers to openlineage client in airflow, i see that provision is there for spark integration via these properties spark.openlineage.transport.headers.xyz --> abcdef" + } + ] + } + ] + } + ], + "team": "T01CWUYP5AR", + "thread_ts": "1694907627.974239", + "reply_count": 1, + "reply_users_count": 1, + "latest_reply": "1695156055.877069", + "reply_users": [ + "U02S6F54MAB" + ], + "is_locked": false, + "subscribed": false, + "replies": [ + { + "client_msg_id": "15a8ab15-edb6-47ed-a3dd-ea593601fb80", + "type": "message", + "text": "there’s no out-of-the-box possibility to do that yet, you’re very welcome to create an issue in GitHub and _maybe_ contribute as well! :slightly_smiling_face:", + "user": "U02S6F54MAB", + "ts": "1695156055.877069", + "blocks": [ + { + "type": "rich_text", + "block_id": "y/Qm5", + "elements": [ + { + "type": "rich_text_section", + "elements": [ + { + "type": "text", + "text": "there’s no out-of-the-box possibility to do that yet, you’re very welcome to create an issue in GitHub and " + }, + { + "type": "text", + "text": "maybe", + "style": { + "italic": true + } + }, + { + "type": "text", + "text": " contribute as well! " + }, + { + "type": "emoji", + "name": "slightly_smiling_face", + "unicode": "1f642" + } + ] + } + ] + } + ], + "team": "T01CWUYP5AR", + "thread_ts": "1694907627.974239", + "parent_user_id": "U05QL7LN2GH" + } + ] + }, + { + "client_msg_id": "8fdf10a3-fcbd-45ca-aea9-e5213e874e13", + "type": "message", + "text": " we have dataproc operator getting called from a dag which submits a spark job, we wanted to maintain that continuity of parent job in the spark job so according to the documentation we can acheive that by using a macro called lineage_run_id that requires task and task_instance as the parameters. The problem we are facing is that our client’s have 1000's of dags, so asking them to change this everywhere it is used is not feasible, so we thought of using the task_policy feature in the airflow…but the problem is that task_policy gives you access to only the task/operator, but we don’t have the access to the task instance..that is required as a parameter to the lineage_run_id function. Can anyone kindly help us on how should we go about this one\n```t1 = DataProcPySparkOperator(\n task_id=job_name,\n #required pyspark configuration,\n job_name=job_name,\n dataproc_pyspark_properties={\n 'spark.driver.extraJavaOptions':\n f\"-javaagent:{jar}={os.environ.get('OPENLINEAGE_URL')}/api/v1/namespaces/{os.getenv('OPENLINEAGE_NAMESPACE', 'default')}/jobs/{job_name}/runs/{{{{macros.OpenLineagePlugin.lineage_run_id(task, task_instance)}}}}?api_key={os.environ.get('OPENLINEAGE_API_KEY')}\"\n dag=dag)```", + "user": "U05QL7LN2GH", + "ts": "1694849427.228709", + "blocks": [ + { + "type": "rich_text", + "block_id": "AHZek", + "elements": [ + { + "type": "rich_text_section", + "elements": [ + { + "type": "broadcast", + "range": "here" + }, + { + "type": "text", + "text": " we have dataproc operator getting called from a dag which submits a spark job, we wanted to maintain that continuity of parent job in the spark job so according to the documentation we can acheive that by using a macro called lineage_run_id that requires task and task_instance as the parameters. The problem we are facing is that our client’s have 1000's of dags, so asking them to change this everywhere it is used is not feasible, so we thought of using the task_policy feature in the airflow…but the problem is that task_policy gives you access to only the task/operator, but we don’t have the access to the task instance..that is required as a parameter to the lineage_run_id function. Can anyone kindly help us on how should we go about this one\n" + } + ] + }, + { + "type": "rich_text_preformatted", + "elements": [ + { + "type": "text", + "text": "t1 = DataProcPySparkOperator(\n task_id=job_name,\n #required pyspark configuration,\n job_name=job_name,\n dataproc_pyspark_properties={\n 'spark.driver.extraJavaOptions':\n f\"-javaagent:{jar}={os.environ.get('OPENLINEAGE_URL')}/api/v1/namespaces/{os.getenv('OPENLINEAGE_NAMESPACE', 'default')}/jobs/{job_name}/runs/{{{{macros.OpenLineagePlugin.lineage_run_id(task, task_instance)}}}}?api_key={os.environ.get('OPENLINEAGE_API_KEY')}\"\n dag=dag)" + } + ], + "border": 0 + } + ] + } + ], + "team": "T01CWUYP5AR", + "edited": { + "user": "U05QL7LN2GH", + "ts": "1694849476.000000" + }, + "thread_ts": "1694849427.228709", + "reply_count": 6, + "reply_users_count": 3, + "latest_reply": "1694854824.785559", + "reply_users": [ + "U02S6F54MAB", + "U05QL7LN2GH", + "U01RA9B5GG2" + ], + "is_locked": false, + "subscribed": false, + "reactions": [ + { + "name": "heavy_plus_sign", + "users": [ + "U05HBLE7YPL" + ], + "count": 1 + } + ], + "replies": [ + { + "client_msg_id": "c74584ec-debc-402b-b867-4cb91564bc1a", + "type": "message", + "text": "you don't need actual task instance to do that. you only should set additional argument as jinja template, same as above", + "user": "U02S6F54MAB", + "ts": "1694852567.185479", + "blocks": [ + { + "type": "rich_text", + "block_id": "YJIIN", + "elements": [ + { + "type": "rich_text_section", + "elements": [ + { + "type": "text", + "text": "you don't need actual task instance to do that. you only should set additional argument as jinja template, same as above" + } + ] + } + ] + } + ], + "team": "T01CWUYP5AR", + "thread_ts": "1694849427.228709", + "parent_user_id": "U05QL7LN2GH" + }, + { + "client_msg_id": "739d43fe-6091-4a33-9e38-49c251a2092f", + "type": "message", + "text": "task_instance in this case is just part of string which is evaluated when jinja render happens", + "user": "U02S6F54MAB", + "ts": "1694852728.566109", + "blocks": [ + { + "type": "rich_text", + "block_id": "JReNS", + "elements": [ + { + "type": "rich_text_section", + "elements": [ + { + "type": "text", + "text": "task_instance in this case is just part of string which is evaluated when jinja render happens" + } + ] + } + ] + } + ], + "team": "T01CWUYP5AR", + "edited": { + "user": "U02S6F54MAB", + "ts": "1694852748.000000" + }, + "thread_ts": "1694849427.228709", + "parent_user_id": "U05QL7LN2GH" + }, + { + "client_msg_id": "fd9ff2ac-d30d-44fe-be46-9fc6bd92cbec", + "type": "message", + "text": "ohh…then we could use the same example as above inside the task_policy to intercept the Operator and add the openlineage specific additions properties?", + "user": "U05QL7LN2GH", + "ts": "1694852830.195709", + "blocks": [ + { + "type": "rich_text", + "block_id": "k9v8D", + "elements": [ + { + "type": "rich_text_section", + "elements": [ + { + "type": "text", + "text": "ohh…then we could use the same example as above inside the task_policy to intercept the Operator and add the openlineage specific additions properties?" + } + ] + } + ] + } + ], + "team": "T01CWUYP5AR", + "thread_ts": "1694849427.228709", + "parent_user_id": "U05QL7LN2GH" + }, + { + "client_msg_id": "7016edd8-d837-4f45-9c0d-baf91cac0447", + "type": "message", + "text": "correct, just remember not to override all properties, just add ol specific", + "user": "U02S6F54MAB", + "ts": "1694853059.395789", + "blocks": [ + { + "type": "rich_text", + "block_id": "4HvJl", + "elements": [ + { + "type": "rich_text_section", + "elements": [ + { + "type": "text", + "text": "correct, just remember not to override all properties, just add ol specific" + } + ] + } + ] + } + ], + "team": "T01CWUYP5AR", + "thread_ts": "1694849427.228709", + "parent_user_id": "U05QL7LN2GH" + }, + { + "client_msg_id": "b03d0357-c247-4634-b6f8-e3456fd93bc5", + "type": "message", + "text": "yeah sure…thank you so much <@U02S6F54MAB>, will try this out and keep you posted", + "user": "U05QL7LN2GH", + "ts": "1694853122.539399", + "blocks": [ + { + "type": "rich_text", + "block_id": "Xsbel", + "elements": [ + { + "type": "rich_text_section", + "elements": [ + { + "type": "text", + "text": "yeah sure…thank you so much " + }, + { + "type": "user", + "user_id": "U02S6F54MAB" + }, + { + "type": "text", + "text": ", will try this out and keep you posted" + } + ] + } + ] + } + ], + "team": "T01CWUYP5AR", + "thread_ts": "1694849427.228709", + "parent_user_id": "U05QL7LN2GH", + "reactions": [ + { + "name": "+1", + "users": [ + "U02S6F54MAB" + ], + "count": 1 + } + ] + }, + { + "client_msg_id": "9e632ff3-3d24-48e1-9b39-dc353db41164", + "type": "message", + "text": "We want to automate setting those options at some point inside the operator itself", + "user": "U01RA9B5GG2", + "ts": "1694854824.785559", + "blocks": [ + { + "type": "rich_text", + "block_id": "pXEbp", + "elements": [ + { + "type": "rich_text_section", + "elements": [ + { + "type": "text", + "text": "We want to automate setting those options at some point inside the operator itself" + } + ] + } + ] + } + ], + "team": "T01CWUYP5AR", + "thread_ts": "1694849427.228709", + "parent_user_id": "U05QL7LN2GH", + "reactions": [ + { + "name": "heavy_plus_sign", + "users": [ + "U05QL7LN2GH" + ], + "count": 1 + } + ] + } + ] + }, + { + "client_msg_id": "f8a7a53a-d531-490e-a231-7e9f83a9f7a3", + "type": "message", + "text": "\nFriendly reminder: the next OpenLineage meetup, our first in Toronto, is happening this coming Monday at 5 PM ET ", + "user": "U02LXF3HUN7", + "ts": "1694793807.376729", + "blocks": [ + { + "type": "rich_text", + "block_id": "liTY7", + "elements": [ + { + "type": "rich_text_section", + "elements": [ + { + "type": "broadcast", + "range": "channel" + }, + { + "type": "text", + "text": "\nFriendly reminder: the next OpenLineage meetup, our first in Toronto, is happening this coming Monday at 5 PM ET " + }, + { + "type": "link", + "url": "https://openlineage.slack.com/archives/C01CK9T7HKR/p1694441261486759" + } + ] + } + ] + } + ], + "team": "T01CWUYP5AR", + "attachments": [ + { + "from_url": "https://openlineage.slack.com/archives/C01CK9T7HKR/p1694441261486759", + "ts": "1694441261.486759", + "author_id": "U02LXF3HUN7", + "channel_id": "C01CK9T7HKR", + "channel_team": "T01CWUYP5AR", + "is_msg_unfurl": true, + "message_blocks": [ + { + "team": "T01CWUYP5AR", + "channel": "C01CK9T7HKR", + "ts": "1694441261.486759", + "message": { + "blocks": [ + { + "type": "rich_text", + "block_id": "t94g1", + "elements": [ + { + "type": "rich_text_section", + "elements": [ + { + "type": "broadcast", + "range": "channel" + }, + { + "type": "text", + "text": "\nThe first Toronto OpenLineage Meetup, featuring a presentation by recent adopter " + }, + { + "type": "link", + "url": "https://metaphor.io/", + "text": "Metaphor" + }, + { + "type": "text", + "text": ", is just one week away. On the agenda:\n" + } + ] + }, + { + "type": "rich_text_list", + "elements": [ + { + "type": "rich_text_section", + "elements": [ + { + "type": "text", + "text": "Evolution of spec presentation/discussion (project background/history)", + "style": { + "bold": true + } + } + ] + }, + { + "type": "rich_text_section", + "elements": [ + { + "type": "text", + "text": "State of the community", + "style": { + "bold": true + } + } + ] + }, + { + "type": "rich_text_section", + "elements": [ + { + "type": "text", + "text": "Integrating OpenLineage with ", + "style": { + "bold": true + } + }, + { + "type": "link", + "url": "https://metaphor.io/", + "text": "Metaphor", + "style": { + "bold": true + } + }, + { + "type": "text", + "text": " (by special guests ", + "style": { + "bold": true + } + }, + { + "type": "link", + "url": "https://www.linkedin.com/in/yeliu84/", + "text": "Ye", + "style": { + "bold": true + } + }, + { + "type": "text", + "text": " & ", + "style": { + "bold": true + } + }, + { + "type": "link", + "url": "https://www.linkedin.com/in/ivanperepelitca/", + "text": "Ivan", + "style": { + "bold": true + } + }, + { + "type": "text", + "text": ")", + "style": { + "bold": true + } + } + ] + }, + { + "type": "rich_text_section", + "elements": [ + { + "type": "text", + "text": "Spark/Column lineage update", + "style": { + "bold": true + } + } + ] + }, + { + "type": "rich_text_section", + "elements": [ + { + "type": "text", + "text": "Airflow Provider update", + "style": { + "bold": true + } + } + ] + }, + { + "type": "rich_text_section", + "elements": [ + { + "type": "text", + "text": "Roadmap Discussion", + "style": { + "bold": true + } + } + ] + } + ], + "style": "ordered", + "indent": 0, + "border": 0 + }, + { + "type": "rich_text_section", + "elements": [ + { + "type": "text", + "text": "Find more details and RSVP ", + "style": { + "bold": true + } + }, + { + "type": "link", + "url": "https://www.meetup.com/openlineage/events/295488014/?utm_medium=referral&utm_campaign=share-btn_savedevents_share_modal&utm_source=link", + "text": "here", + "style": { + "bold": true + } + }, + { + "type": "text", + "text": "." + } + ] + } + ] + } + ] + } + } + ], + "id": 1, + "original_url": "https://openlineage.slack.com/archives/C01CK9T7HKR/p1694441261486759", + "fallback": "[September 11th, 2023 7:07 AM] michael282: \nThe first Toronto OpenLineage Meetup, featuring a presentation by recent adopter , is just one week away. On the agenda:\n1. *Evolution of spec presentation/discussion (project background/history)*\n2. *State of the community*\n3. *Integrating OpenLineage with (by special guests & )*\n4. *Spark/Column lineage update*\n5. *Airflow Provider update*\n6. *Roadmap Discussion*\n*Find more details and RSVP *.", + "text": "\nThe first Toronto OpenLineage Meetup, featuring a presentation by recent adopter , is just one week away. On the agenda:\n1. *Evolution of spec presentation/discussion (project background/history)*\n2. *State of the community*\n3. *Integrating OpenLineage with (by special guests & )*\n4. *Spark/Column lineage update*\n5. *Airflow Provider update*\n6. *Roadmap Discussion*\n*Find more details and RSVP *.", + "author_name": "Michael Robinson", + "author_link": "https://openlineage.slack.com/team/U02LXF3HUN7", + "author_icon": "https://avatars.slack-edge.com/2022-01-25/3019716733729_66fea720e9504dc08144_48.jpg", + "author_subname": "Michael Robinson", + "mrkdwn_in": [ + "text" + ], + "footer": "Slack Conversation" + } + ], + "reactions": [ + { + "name": "+1", + "users": [ + "U01RA9B5GG2" + ], + "count": 1 + } + ] + }, + { + "client_msg_id": "32e40b58-9b35-45a3-99aa-e37404cd6329", + "type": "message", + "text": "Per discussion in the OpenLineage sync today here is a very early strawman proposal for an OpenLineage registry that producers and consumers could be registered in.\nFeedback or alternate proposals welcome\n\nOnce this is sufficiently fleshed out, I’ll create an actual proposal on github", + "user": "U01DCLP0GU9", + "ts": "1694737381.437569", + "blocks": [ + { + "type": "rich_text", + "block_id": "KKjtL", + "elements": [ + { + "type": "rich_text_section", + "elements": [ + { + "type": "text", + "text": "Per discussion in the OpenLineage sync today here is a very early strawman proposal for an OpenLineage registry that producers and consumers could be registered in.\nFeedback or alternate proposals welcome\n" + }, + { + "type": "link", + "url": "https://docs.google.com/document/d/1zIxKST59q3I6ws896M4GkUn7IsueLw8ejct5E-TR0vY/edit" + }, + { + "type": "text", + "text": "\nOnce this is sufficiently fleshed out, I’ll create an actual proposal on github" + } + ] + } + ] + } + ], + "team": "T01CWUYP5AR", + "thread_ts": "1694737381.437569", + "reply_count": 2, + "reply_users_count": 1, + "latest_reply": "1696541652.452819", + "reply_users": [ + "U01DCLP0GU9" + ], + "is_locked": false, + "subscribed": false, + "reactions": [ + { + "name": "+1", + "users": [ + "U01RA9B5GG2" + ], + "count": 1 + } + ], + "replies": [ + { + "type": "message", + "subtype": "thread_broadcast", + "text": "I have cleaned up the registry proposal.\n\nIn particular:\n• I clarified that option 2 is preferred at this point.\n• I moved discussion notes to the bottom. they will go away at some point\n• Once it is stable, I’ll create a with the preferred option.\n• we need a good proposal for the core facets prefix. My suggestion is to move core facets to `core` in the registry. The drawback is prefix would be inconsistent.\n", + "user": "U01DCLP0GU9", + "ts": "1696379615.265919", + "thread_ts": "1694737381.437569", + "root": { + "client_msg_id": "32e40b58-9b35-45a3-99aa-e37404cd6329", + "type": "message", + "text": "Per discussion in the OpenLineage sync today here is a very early strawman proposal for an OpenLineage registry that producers and consumers could be registered in.\nFeedback or alternate proposals welcome\n\nOnce this is sufficiently fleshed out, I’ll create an actual proposal on github", + "user": "U01DCLP0GU9", + "ts": "1694737381.437569", + "blocks": [ + { + "type": "rich_text", + "block_id": "KKjtL", + "elements": [ + { + "type": "rich_text_section", + "elements": [ + { + "type": "text", + "text": "Per discussion in the OpenLineage sync today here is a very early strawman proposal for an OpenLineage registry that producers and consumers could be registered in.\nFeedback or alternate proposals welcome\n" + }, + { + "type": "link", + "url": "https://docs.google.com/document/d/1zIxKST59q3I6ws896M4GkUn7IsueLw8ejct5E-TR0vY/edit" + }, + { + "type": "text", + "text": "\nOnce this is sufficiently fleshed out, I’ll create an actual proposal on github" + } + ] + } + ] + } + ], + "team": "T01CWUYP5AR", + "thread_ts": "1694737381.437569", + "reply_count": 2, + "reply_users_count": 1, + "latest_reply": "1696541652.452819", + "reply_users": [ + "U01DCLP0GU9" + ], + "is_locked": false, + "subscribed": false + }, + "blocks": [ + { + "type": "rich_text", + "block_id": "UD6d9", + "elements": [ + { + "type": "rich_text_section", + "elements": [ + { + "type": "text", + "text": "I have cleaned up the registry proposal.\n" + }, + { + "type": "link", + "url": "https://docs.google.com/document/d/1zIxKST59q3I6ws896M4GkUn7IsueLw8ejct5E-TR0vY/edit" + }, + { + "type": "text", + "text": "\nIn particular:\n" + } + ] + }, + { + "type": "rich_text_list", + "elements": [ + { + "type": "rich_text_section", + "elements": [ + { + "type": "text", + "text": "I clarified that option 2 is preferred at this point." + } + ] + }, + { + "type": "rich_text_section", + "elements": [ + { + "type": "text", + "text": "I moved discussion notes to the bottom. they will go away at some point" + } + ] + }, + { + "type": "rich_text_section", + "elements": [ + { + "type": "text", + "text": "Once it is stable, I’ll create a " + }, + { + "type": "link", + "url": "https://github.com/OpenLineage/OpenLineage/tree/main/proposals", + "text": "proposal" + }, + { + "type": "text", + "text": " with the preferred option." + } + ] + }, + { + "type": "rich_text_section", + "elements": [ + { + "type": "text", + "text": "we need a good proposal for the core facets prefix. My suggestion is to move core facets to " + }, + { + "type": "text", + "text": "core", + "style": { + "code": true + } + }, + { + "type": "text", + "text": " in the registry. The drawback is prefix would be inconsistent." + } + ] + } + ], + "style": "bullet", + "indent": 0, + "border": 0 + }, + { + "type": "rich_text_section", + "elements": [] + } + ] + } + ], + "client_msg_id": "4f7a5bb9-269f-4e6d-98b4-669b2760c1bf" + }, + { + "type": "message", + "subtype": "thread_broadcast", + "text": "I have created a ticket to make this easier to find. Once I get more feedback I’ll turn it into a md file in the repo: \n", + "user": "U01DCLP0GU9", + "ts": "1696541652.452819", + "thread_ts": "1694737381.437569", + "root": { + "client_msg_id": "32e40b58-9b35-45a3-99aa-e37404cd6329", + "type": "message", + "text": "Per discussion in the OpenLineage sync today here is a very early strawman proposal for an OpenLineage registry that producers and consumers could be registered in.\nFeedback or alternate proposals welcome\n\nOnce this is sufficiently fleshed out, I’ll create an actual proposal on github", + "user": "U01DCLP0GU9", + "ts": "1694737381.437569", + "blocks": [ + { + "type": "rich_text", + "block_id": "KKjtL", + "elements": [ + { + "type": "rich_text_section", + "elements": [ + { + "type": "text", + "text": "Per discussion in the OpenLineage sync today here is a very early strawman proposal for an OpenLineage registry that producers and consumers could be registered in.\nFeedback or alternate proposals welcome\n" + }, + { + "type": "link", + "url": "https://docs.google.com/document/d/1zIxKST59q3I6ws896M4GkUn7IsueLw8ejct5E-TR0vY/edit" + }, + { + "type": "text", + "text": "\nOnce this is sufficiently fleshed out, I’ll create an actual proposal on github" + } + ] + } + ] + } + ], + "team": "T01CWUYP5AR", + "thread_ts": "1694737381.437569", + "reply_count": 2, + "reply_users_count": 1, + "latest_reply": "1696541652.452819", + "reply_users": [ + "U01DCLP0GU9" + ], + "is_locked": false, + "subscribed": false + }, + "blocks": [ + { + "type": "rich_text", + "block_id": "EbQGP", + "elements": [ + { + "type": "rich_text_section", + "elements": [ + { + "type": "text", + "text": "I have created a ticket to make this easier to find. Once I get more feedback I’ll turn it into a md file in the repo: " + }, + { + "type": "link", + "url": "https://docs.google.com/document/d/1zIxKST59q3I6ws896M4GkUn7IsueLw8ejct5E-TR0vY/edit#heading=h.enpbmvu7n8gu" + }, + { + "type": "text", + "text": "\n" + }, + { + "type": "link", + "url": "https://github.com/OpenLineage/OpenLineage/issues/2161" + } + ] + } + ] + } + ], + "attachments": [ + { + "id": 1, + "footer_icon": "https://slack.github.com/static/img/favicon-neutral.png", + "ts": 1696541261, + "color": "36a64f", + "bot_id": "B01VA0FB340", + "app_unfurl_url": "https://github.com/OpenLineage/OpenLineage/issues/2161", + "is_app_unfurl": true, + "app_id": "A01BP7R4KNY", + "fallback": "#2161 [PROPOSAL] Add a Registry of Producers and Consumers in OpenLineage", + "text": "*Purpose*\n\nThis is the early stage of an idea to get community feedback on what an OpenLineage registry for producers, custom facets and consumers could be. Once this document is stable enough, I’ll create an official proposal on the OpenLineage repo.\n\n*Goal*\n\nAllow third parties to register their implementations or custom extensions to make them easy to discover. \nShorten “Producer” and “schema url” values\n\n*Proposed implementation*\n\nCurrent draft for discussion:\n\n", + "title": "#2161 [PROPOSAL] Add a Registry of Producers and Consumers in OpenLineage", + "title_link": "https://github.com/OpenLineage/OpenLineage/issues/2161", + "footer": "", + "fields": [ + { + "value": "proposal", + "title": "Labels", + "short": true + } + ], + "mrkdwn_in": [ + "text" + ] + } + ], + "client_msg_id": "b2b64b49-4c25-427f-b1ae-f1fa92aad028", + "edited": { + "user": "U01DCLP0GU9", + "ts": "1696541673.000000" + } + } + ] + }, + { + "client_msg_id": "64406ba1-f8a0-403e-8819-6a11eddd1ec5", + "type": "message", + "text": "Hey everyone,\nAny chance we could have a *openlineage-integration-common* 1.1.1 release with the following changes..?\n• \n• ", + "user": "U055N2GRT4P", + "ts": "1694700221.242579", + "blocks": [ + { + "type": "rich_text", + "block_id": "KjDBh", + "elements": [ + { + "type": "rich_text_section", + "elements": [ + { + "type": "text", + "text": "Hey everyone,\nAny chance we could have a " + }, + { + "type": "text", + "text": "openlineage-integration-common", + "style": { + "bold": true + } + }, + { + "type": "text", + "text": " 1.1.1 release with the following changes..?\n" + } + ] + }, + { + "type": "rich_text_list", + "elements": [ + { + "type": "rich_text_section", + "elements": [ + { + "type": "link", + "url": "https://github.com/OpenLineage/OpenLineage/pull/2106" + } + ] + }, + { + "type": "rich_text_section", + "elements": [ + { + "type": "link", + "url": "https://github.com/OpenLineage/OpenLineage/pull/2108" + } + ] + } + ], + "style": "bullet", + "indent": 0, + "border": 0 + } + ] + } + ], + "team": "T01CWUYP5AR", + "thread_ts": "1694700221.242579", + "reply_count": 4, + "reply_users_count": 2, + "latest_reply": "1694767212.306929", + "reply_users": [ + "U055N2GRT4P", + "U02LXF3HUN7" + ], + "is_locked": false, + "subscribed": true, + "last_read": "1694767212.306929", + "reactions": [ + { + "name": "heavy_plus_sign", + "users": [ + "U02LXF3HUN7", + "U01HNKK4XAM", + "U01RA9B5GG2", + "U02S6F54MAB", + "U02MK6YNAQ5", + "U01DCLP0GU9" + ], + "count": 6 + } + ], + "replies": [ + { + "client_msg_id": "8b7ef37c-6527-4a9f-a021-b7603fc1b02a", + "type": "message", + "text": "Specially the first PR is affecting users of the *astronomer-cosmos* library: ", + "user": "U055N2GRT4P", + "ts": "1694700319.444869", + "blocks": [ + { + "type": "rich_text", + "block_id": "oAc8+", + "elements": [ + { + "type": "rich_text_section", + "elements": [ + { + "type": "text", + "text": "Specially the first PR is affecting users of the " + }, + { + "type": "text", + "text": "astronomer-cosmos", + "style": { + "bold": true + } + }, + { + "type": "text", + "text": " library: " + }, + { + "type": "link", + "url": "https://github.com/astronomer/astronomer-cosmos/issues/533" + } + ] + } + ] + } + ], + "team": "T01CWUYP5AR", + "thread_ts": "1694700221.242579", + "parent_user_id": "U055N2GRT4P" + }, + { + "client_msg_id": "518c0176-d673-4505-b728-c1971a00c897", + "type": "message", + "text": "Thanks <@U055N2GRT4P> for requesting your first OpenLineage release! Three +1s from committers will authorize", + "user": "U02LXF3HUN7", + "ts": "1694700324.134709", + "blocks": [ + { + "type": "rich_text", + "block_id": "g+12P", + "elements": [ + { + "type": "rich_text_section", + "elements": [ + { + "type": "text", + "text": "Thanks " + }, + { + "type": "user", + "user_id": "U055N2GRT4P" + }, + { + "type": "text", + "text": " for requesting your first OpenLineage release! Three +1s from committers will authorize" + } + ] + } + ] + } + ], + "team": "T01CWUYP5AR", + "thread_ts": "1694700221.242579", + "parent_user_id": "U055N2GRT4P", + "reactions": [ + { + "name": "gratitude-thank-you", + "users": [ + "U055N2GRT4P" + ], + "count": 1 + } + ] + }, + { + "client_msg_id": "cb4da1c3-9374-42ee-a162-5c7937d788d8", + "type": "message", + "text": "The release is authorized and will be initiated within two business days.", + "user": "U02LXF3HUN7", + "ts": "1694707195.166749", + "blocks": [ + { + "type": "rich_text", + "block_id": "iREVS", + "elements": [ + { + "type": "rich_text_section", + "elements": [ + { + "type": "text", + "text": "The release is authorized and will be initiated within two business days." + } + ] + } + ] + } + ], + "team": "T01CWUYP5AR", + "thread_ts": "1694700221.242579", + "parent_user_id": "U055N2GRT4P", + "reactions": [ + { + "name": "tada", + "users": [ + "U055N2GRT4P" + ], + "count": 1 + } + ] + }, + { + "client_msg_id": "d076f666-77f2-4e04-9f84-d5a066b8a7ed", + "type": "message", + "text": "Thanks a lot, <@U02LXF3HUN7>!", + "user": "U055N2GRT4P", + "ts": "1694767212.306929", + "blocks": [ + { + "type": "rich_text", + "block_id": "gbUl/", + "elements": [ + { + "type": "rich_text_section", + "elements": [ + { + "type": "text", + "text": "Thanks a lot, " + }, + { + "type": "user", + "user_id": "U02LXF3HUN7" + }, + { + "type": "text", + "text": "!" + } + ] + } + ] + } + ], + "team": "T01CWUYP5AR", + "thread_ts": "1694700221.242579", + "parent_user_id": "U055N2GRT4P" + } + ] + }, + { + "client_msg_id": "09f3740d-1573-4bcc-9ef8-d980019a3fcd", + "type": "message", + "text": "Context:\n\nWe use Spark with YARN, running on Hadoop 2.x (I can't remember the exact minor version) with Hive support.\n\nProblem:\n\nI'm noticed that `CreateDataSourceAsSelectCommand` objects are _always_ transformed to an `OutputDataset` with a _namespace_ value set to `file` - which is curious, because the inputs always have a (correct) namespace of `hdfs://<name-node>` - is this a known issue? A flaw with Apache Spark? A bug in the resolution logic?\n\nFor reference:\n\n```public class CreateDataSourceTableCommandVisitor\n extends QueryPlanVisitor<CreateDataSourceTableCommand, OpenLineage.OutputDataset> {\n\n public CreateDataSourceTableCommandVisitor(OpenLineageContext context) {\n super(context);\n }\n\n @Override\n public List<OpenLineage.OutputDataset> apply(LogicalPlan x) {\n CreateDataSourceTableCommand command = (CreateDataSourceTableCommand) x;\n CatalogTable catalogTable = command.table();\n\n return Collections.singletonList(\n outputDataset()\n .getDataset(\n PathUtils.fromCatalogTable(catalogTable),\n catalogTable.schema(),\n OpenLineage.LifecycleStateChangeDatasetFacet.LifecycleStateChange.CREATE));\n }\n}```\nRunning this: `cat events.log | jq '{eventTime: .eventTime, eventType: .eventType, runId: .run.runId, jobNamespace: .job.namespace, jobName: .job.name, outputs: .outputs[] | {namespace: .namespace, name: .name}, inputs: .inputs[] | {namespace: .namespace, name: .name}}'`\n\nThis is an output:\n```{\n \"eventTime\": \"2023-09-13T16:01:27.059Z\",\n \"eventType\": \"START\",\n \"runId\": \"bbbb5763-3615-46c0-95ca-1fc398c91d5d\",\n \"jobNamespace\": \"spark.cluster-1\",\n \"jobName\": \"ol_hadoop_test.execute_create_data_source_table_as_select_command.dhawes_db_ol_test_hadoop_tgt\",\n \"outputs\": {\n \"namespace\": \"file\",\n \"name\": \"/user/hive/warehouse/dhawes.db/ol_test_hadoop_tgt\"\n },\n \"inputs\": {\n \"namespace\": \"\",\n \"name\": \"/user/hive/warehouse/dhawes.db/ol_test_hadoop_src\"\n }\n}```", + "user": "U05FLJE4GDU", + "ts": "1694686815.337029", + "blocks": [ + { + "type": "rich_text", + "block_id": "1OX1T", + "elements": [ + { + "type": "rich_text_section", + "elements": [ + { + "type": "text", + "text": "Context:\n\nWe use Spark with YARN, running on Hadoop 2.x (I can't remember the exact minor version) with Hive support.\n\nProblem:\n\nI'm noticed that " + }, + { + "type": "text", + "text": "CreateDataSourceAsSelectCommand", + "style": { + "code": true + } + }, + { + "type": "text", + "text": " objects are " + }, + { + "type": "text", + "text": "always", + "style": { + "italic": true + } + }, + { + "type": "text", + "text": " transformed to an " + }, + { + "type": "text", + "text": "OutputDataset", + "style": { + "code": true + } + }, + { + "type": "text", + "text": " with a " + }, + { + "type": "text", + "text": "namespace", + "style": { + "italic": true + } + }, + { + "type": "text", + "text": " value set to " + }, + { + "type": "text", + "text": "file", + "style": { + "code": true + } + }, + { + "type": "text", + "text": " - which is curious, because the inputs always have a (correct) namespace of " + }, + { + "type": "text", + "text": "hdfs://", + "style": { + "code": true + } + }, + { + "type": "text", + "text": " - is this a known issue? A flaw with Apache Spark? A bug in the resolution logic?\n\nFor reference:\n\n" + } + ] + }, + { + "type": "rich_text_preformatted", + "elements": [ + { + "type": "text", + "text": "public class CreateDataSourceTableCommandVisitor\n extends QueryPlanVisitor {\n\n public CreateDataSourceTableCommandVisitor(OpenLineageContext context) {\n super(context);\n }\n\n @Override\n public List apply(LogicalPlan x) {\n CreateDataSourceTableCommand command = (CreateDataSourceTableCommand) x;\n CatalogTable catalogTable = command.table();\n\n return Collections.singletonList(\n outputDataset()\n .getDataset(\n PathUtils.fromCatalogTable(catalogTable),\n catalogTable.schema(),\n OpenLineage.LifecycleStateChangeDatasetFacet.LifecycleStateChange.CREATE));\n }\n}" + } + ], + "border": 0 + }, + { + "type": "rich_text_section", + "elements": [ + { + "type": "text", + "text": "\nRunning this: " + }, + { + "type": "text", + "text": "cat events.log | jq '{eventTime: .eventTime, eventType: .eventType, runId: .run.runId, jobNamespace: .job.namespace, jobName: .job.name, outputs: .outputs[] | {namespace: .namespace, name: .name}, inputs: .inputs[] | {namespace: .namespace, name: .name}}'", + "style": { + "code": true + } + }, + { + "type": "text", + "text": "\n\nThis is an output:\n" + } + ] + }, + { + "type": "rich_text_preformatted", + "elements": [ + { + "type": "text", + "text": "{\n \"eventTime\": \"2023-09-13T16:01:27.059Z\",\n \"eventType\": \"START\",\n \"runId\": \"bbbb5763-3615-46c0-95ca-1fc398c91d5d\",\n \"jobNamespace\": \"spark.cluster-1\",\n \"jobName\": \"ol_hadoop_test.execute_create_data_source_table_as_select_command.dhawes_db_ol_test_hadoop_tgt\",\n \"outputs\": {\n \"namespace\": \"file\",\n \"name\": \"/user/hive/warehouse/dhawes.db/ol_test_hadoop_tgt\"\n },\n \"inputs\": {\n \"namespace\": \"" + }, + { + "type": "link", + "url": "hdfs://nn1" + }, + { + "type": "text", + "text": "\",\n \"name\": \"/user/hive/warehouse/dhawes.db/ol_test_hadoop_src\"\n }\n}" + } + ], + "border": 0 + } + ] + } + ], + "team": "T01CWUYP5AR", + "edited": { + "user": "U05FLJE4GDU", + "ts": "1694687550.000000" + }, + "thread_ts": "1694686815.337029", + "reply_count": 3, + "reply_users_count": 2, + "latest_reply": "1694697839.386649", + "reply_users": [ + "U02MK6YNAQ5", + "U05FLJE4GDU" + ], + "is_locked": false, + "subscribed": false, + "reactions": [ + { + "name": "eyes", + "users": [ + "U02MK6YNAQ5" + ], + "count": 1 + } + ], + "replies": [ + { + "client_msg_id": "67530b68-4b7d-42d2-8bd9-f207888169d2", + "type": "message", + "text": "Seems like an issue on our side. Do you know how the source is read? What LogicalPlan leaf is used to read src? Would love to find how is this done differently", + "user": "U02MK6YNAQ5", + "ts": "1694691145.323709", + "blocks": [ + { + "type": "rich_text", + "block_id": "NPu0C", + "elements": [ + { + "type": "rich_text_section", + "elements": [ + { + "type": "text", + "text": "Seems like an issue on our side. Do you know how the source is read? What LogicalPlan leaf is used to read src? Would love to find how is this done differently" + } + ] + } + ] + } + ], + "team": "T01CWUYP5AR", + "thread_ts": "1694686815.337029", + "parent_user_id": "U05FLJE4GDU" + }, + { + "client_msg_id": "4f3db9e2-2137-478c-86e0-2a4d9d7dad6f", + "type": "message", + "text": "Hmm, I'll have to do explain plan to see what exactly it is.\n\nHowever my sample job uses `spark.sql(\"SELECT * FROM dhawes.ol_test_hadoop_src\")`\n\nwhich itself is created using\n\n```spark.sql(\"SELECT 1 AS id\").write.format(\"orc\").mode(\"overwrite\").saveAsTable(\"dhawes.ol_test_hadoop_src\")```\n", + "user": "U05FLJE4GDU", + "ts": "1694697418.142389", + "blocks": [ + { + "type": "rich_text", + "block_id": "zWFes", + "elements": [ + { + "type": "rich_text_section", + "elements": [ + { + "type": "text", + "text": "Hmm, I'll have to do explain plan to see what exactly it is.\n\nHowever my sample job uses " + }, + { + "type": "text", + "text": "spark.sql(\"SELECT * FROM dhawes.ol_test_hadoop_src\")", + "style": { + "code": true + } + }, + { + "type": "text", + "text": "\n\nwhich itself is created using\n\n" + } + ] + }, + { + "type": "rich_text_preformatted", + "elements": [ + { + "type": "text", + "text": "spark.sql(\"SELECT 1 AS id\").write.format(\"orc\").mode(\"overwrite\").saveAsTable(\"dhawes.ol_test_hadoop_src\")" + } + ], + "border": 0 + }, + { + "type": "rich_text_section", + "elements": [] + } + ] + } + ], + "team": "T01CWUYP5AR", + "thread_ts": "1694686815.337029", + "parent_user_id": "U05FLJE4GDU" + }, + { + "client_msg_id": "bde1a4fd-2425-4d60-abcd-40d6e55044a0", + "type": "message", + "text": "```>>> spark.sql(\"SELECT * FROM dhawes.ol_test_hadoop_src\").explain(True)\n== Parsed Logical Plan ==\n'Project [*]\n+- 'UnresolvedRelation `dhawes`.`ol_test_hadoop_src`\n\n== Analyzed Logical Plan ==\nid: int\nProject [id#3]\n+- SubqueryAlias `dhawes`.`ol_test_hadoop_src`\n +- Relation[id#3] orc\n\n== Optimized Logical Plan ==\nRelation[id#3] orc\n\n== Physical Plan ==\n*(1) FileScan orc dhawes.ol_test_hadoop_src[id#3] Batched: true, Format: ORC, Location: InMemoryFileIndex[], PartitionFilters: [], PushedFilters: [], ReadSchema: struct<id:int>```\n", + "user": "U05FLJE4GDU", + "ts": "1694697839.386649", + "blocks": [ + { + "type": "rich_text", + "block_id": "HMpuD", + "elements": [ + { + "type": "rich_text_preformatted", + "elements": [ + { + "type": "text", + "text": ">>> spark.sql(\"SELECT * FROM dhawes.ol_test_hadoop_src\").explain(True)\n== Parsed Logical Plan ==\n'Project [*]\n+- 'UnresolvedRelation `dhawes`.`ol_test_hadoop_src`\n\n== Analyzed Logical Plan ==\nid: int\nProject [id#3]\n+- SubqueryAlias `dhawes`.`ol_test_hadoop_src`\n +- Relation[id#3] orc\n\n== Optimized Logical Plan ==\nRelation[id#3] orc\n\n== Physical Plan ==\n*(1) FileScan orc dhawes.ol_test_hadoop_src[id#3] Batched: true, Format: ORC, Location: InMemoryFileIndex[" + }, + { + "type": "link", + "url": "hdfs://nn1/user/hive/warehouse/dhawes.db/ol_test_hadoop_src" + }, + { + "type": "text", + "text": "], PartitionFilters: [], PushedFilters: [], ReadSchema: struct" + } + ], + "border": 0 + }, + { + "type": "rich_text_section", + "elements": [] + } + ] + } + ], + "team": "T01CWUYP5AR", + "thread_ts": "1694686815.337029", + "parent_user_id": "U05FLJE4GDU" + } + ] + }, + { + "client_msg_id": "e3ce0cb2-d60c-4e67-b0d1-ca22551c75aa", + "type": "message", + "text": "\nThis month’s TSC meeting, open to all, is tomorrow: ", + "user": "U02LXF3HUN7", + "ts": "1694629232.934029", + "blocks": [ + { + "type": "rich_text", + "block_id": "q4mW7", + "elements": [ + { + "type": "rich_text_section", + "elements": [ + { + "type": "broadcast", + "range": "channel" + }, + { + "type": "text", + "text": "\nThis month’s TSC meeting, open to all, is tomorrow: " + }, + { + "type": "link", + "url": "https://openlineage.slack.com/archives/C01CK9T7HKR/p1694113940400549" + } + ] + } + ] + } + ], + "team": "T01CWUYP5AR", + "attachments": [ + { + "from_url": "https://openlineage.slack.com/archives/C01CK9T7HKR/p1694113940400549", + "ts": "1694113940.400549", + "author_id": "U02LXF3HUN7", + "channel_id": "C01CK9T7HKR", + "channel_team": "T01CWUYP5AR", + "is_msg_unfurl": true, + "message_blocks": [ + { + "team": "T01CWUYP5AR", + "channel": "C01CK9T7HKR", + "ts": "1694113940.400549", + "message": { + "blocks": [ + { + "type": "rich_text", + "block_id": "Yv9ts", + "elements": [ + { + "type": "rich_text_section", + "elements": [ + { + "type": "broadcast", + "range": "channel" + }, + { + "type": "text", + "text": "\nThis month’s TSC meeting is next Thursday the 14th at 10am PT. On the tentative agenda:\n" + } + ] + }, + { + "type": "rich_text_list", + "elements": [ + { + "type": "rich_text_section", + "elements": [ + { + "type": "text", + "text": "announcements" + } + ] + }, + { + "type": "rich_text_section", + "elements": [ + { + "type": "text", + "text": "recent releases" + } + ] + }, + { + "type": "rich_text_section", + "elements": [ + { + "type": "text", + "text": "demo: Spark integration tests in Databricks runtime" + } + ] + }, + { + "type": "rich_text_section", + "elements": [ + { + "type": "text", + "text": "open discussion" + } + ] + }, + { + "type": "rich_text_section", + "elements": [ + { + "type": "text", + "text": "more (TBA)" + } + ] + } + ], + "style": "bullet", + "indent": 0, + "border": 0 + }, + { + "type": "rich_text_section", + "elements": [ + { + "type": "text", + "text": "More info and the meeting link can be found on the " + }, + { + "type": "link", + "url": "https://openlineage.io/meetings/", + "text": "website" + }, + { + "type": "text", + "text": ". All are welcome! Also, feel free to reply or DM me with discussion topics, agenda items, etc." + } + ] + } + ] + } + ] + } + } + ], + "id": 1, + "original_url": "https://openlineage.slack.com/archives/C01CK9T7HKR/p1694113940400549", + "fallback": "[September 7th, 2023 12:12 PM] michael282: \nThis month’s TSC meeting is next Thursday the 14th at 10am PT. On the tentative agenda:\n• announcements\n• recent releases\n• demo: Spark integration tests in Databricks runtime\n• open discussion\n• more (TBA)\nMore info and the meeting link can be found on the . All are welcome! Also, feel free to reply or DM me with discussion topics, agenda items, etc.", + "text": "\nThis month’s TSC meeting is next Thursday the 14th at 10am PT. On the tentative agenda:\n• announcements\n• recent releases\n• demo: Spark integration tests in Databricks runtime\n• open discussion\n• more (TBA)\nMore info and the meeting link can be found on the . All are welcome! Also, feel free to reply or DM me with discussion topics, agenda items, etc.", + "author_name": "Michael Robinson", + "author_link": "https://openlineage.slack.com/team/U02LXF3HUN7", + "author_icon": "https://avatars.slack-edge.com/2022-01-25/3019716733729_66fea720e9504dc08144_48.jpg", + "author_subname": "Michael Robinson", + "mrkdwn_in": [ + "text" + ], + "footer": "Slack Conversation" + } + ], + "reactions": [ + { + "name": "white_check_mark", + "users": [ + "U0323HG8C8H" + ], + "count": 1 + } + ] + }, + { + "client_msg_id": "84e1efcc-8e5c-4334-b18f-5279acee755d", + "type": "message", + "text": "I am exploring Spark - OpenLineage integration (using the latest PySpark and OL versions). I tested a simple pipeline which:\n• Reads JSON data into PySpark DataFrame\n• Apply data transformations\n• Write transformed data to MySQL database\nObserved that we receive 4 events (2 `START` and 2 `COMPLETE`) for the same job name. The events are almost identical with a small diff in the facets. All the events share the same `runId`, and we don't get any `parentRunId`.\nTeam, can you please confirm if this behaviour is expected? Seems to be different from the Airflow integration where we relate jobs to Parent Jobs.", + "user": "U05A1D80QKF", + "ts": "1694583867.900909", + "blocks": [ + { + "type": "rich_text", + "block_id": "p9tSR", + "elements": [ + { + "type": "rich_text_section", + "elements": [ + { + "type": "text", + "text": "I am exploring Spark - OpenLineage integration (using the latest PySpark and OL versions). I tested a simple pipeline which:\n" + } + ] + }, + { + "type": "rich_text_list", + "elements": [ + { + "type": "rich_text_section", + "elements": [ + { + "type": "text", + "text": "Reads JSON data into PySpark DataFrame" + } + ] + }, + { + "type": "rich_text_section", + "elements": [ + { + "type": "text", + "text": "Apply data transformations" + } + ] + }, + { + "type": "rich_text_section", + "elements": [ + { + "type": "text", + "text": "Write transformed data to MySQL database" + } + ] + } + ], + "style": "bullet", + "indent": 0, + "border": 0 + }, + { + "type": "rich_text_section", + "elements": [ + { + "type": "text", + "text": "Observed that we receive 4 events (2 " + }, + { + "type": "text", + "text": "START", + "style": { + "code": true + } + }, + { + "type": "text", + "text": " and 2 " + }, + { + "type": "text", + "text": "COMPLETE", + "style": { + "code": true + } + }, + { + "type": "text", + "text": ") for the same job name. The events are almost identical with a small diff in the facets. All the events share the same " + }, + { + "type": "text", + "text": "runId", + "style": { + "code": true + } + }, + { + "type": "text", + "text": ", and we don't get any " + }, + { + "type": "text", + "text": "parentRunId", + "style": { + "code": true + } + }, + { + "type": "text", + "text": ".\nTeam, can you please confirm if this behaviour is expected? Seems to be different from the Airflow integration where we relate jobs to Parent Jobs." + } + ] + } + ] + } + ], + "team": "T01CWUYP5AR", + "thread_ts": "1694583867.900909", + "reply_count": 16, + "reply_users_count": 4, + "latest_reply": "1695717183.779589", + "reply_users": [ + "U05FLJE4GDU", + "U05A1D80QKF", + "U01RA9B5GG2", + "U02MK6YNAQ5" + ], + "is_locked": false, + "subscribed": false, + "replies": [ + { + "client_msg_id": "1d737c7a-5a41-4d36-87c1-8aecd19ca7ea", + "type": "message", + "text": "The Spark integration requires that two parameters are passed to it, namely:\n\n```spark.openlineage.parentJobName\nspark.openlineage.parentRunId```\nYou can find the list of parameters here:\n\n", + "user": "U05FLJE4GDU", + "ts": "1694588077.143419", + "blocks": [ + { + "type": "rich_text", + "block_id": "0UiGZ", + "elements": [ + { + "type": "rich_text_section", + "elements": [ + { + "type": "text", + "text": "The Spark integration requires that two parameters are passed to it, namely:\n\n" + } + ] + }, + { + "type": "rich_text_preformatted", + "elements": [ + { + "type": "text", + "text": "spark.openlineage.parentJobName\nspark.openlineage.parentRunId" + } + ], + "border": 0 + }, + { + "type": "rich_text_section", + "elements": [ + { + "type": "text", + "text": "You can find the list of parameters here:\n\n" + }, + { + "type": "link", + "url": "https://github.com/OpenLineage/OpenLineage/blob/main/integration/spark/README.md" + } + ] + } + ] + } + ], + "team": "T01CWUYP5AR", + "edited": { + "user": "U05FLJE4GDU", + "ts": "1694588116.000000" + }, + "thread_ts": "1694583867.900909", + "parent_user_id": "U05A1D80QKF" + }, + { + "client_msg_id": "ad9ed647-58e2-45af-8139-274c7904e9e6", + "type": "message", + "text": "Thanks, will check this out", + "user": "U05A1D80QKF", + "ts": "1694588151.594469", + "blocks": [ + { + "type": "rich_text", + "block_id": "lbzzq", + "elements": [ + { + "type": "rich_text_section", + "elements": [ + { + "type": "text", + "text": "Thanks, will check this out" + } + ] + } + ] + } + ], + "team": "T01CWUYP5AR", + "thread_ts": "1694583867.900909", + "parent_user_id": "U05A1D80QKF" + }, + { + "client_msg_id": "a8e782ca-76dc-4f36-9d61-37d9eaebaa86", + "type": "message", + "text": "As for double accounting of events - that's a bit harder to diagnose.", + "user": "U05FLJE4GDU", + "ts": "1694588263.483749", + "blocks": [ + { + "type": "rich_text", + "block_id": "LZeql", + "elements": [ + { + "type": "rich_text_section", + "elements": [ + { + "type": "text", + "text": "As for double accounting of events - that's a bit harder to diagnose." + } + ] + } + ] + } + ], + "team": "T01CWUYP5AR", + "thread_ts": "1694583867.900909", + "parent_user_id": "U05A1D80QKF" + }, + { + "client_msg_id": "4bb93ddd-263b-423e-b260-f2bea978c19f", + "type": "message", + "text": "Can you share the the job and events?\nAlso <@U02MK6YNAQ5>", + "user": "U01RA9B5GG2", + "ts": "1694593983.249679", + "blocks": [ + { + "type": "rich_text", + "block_id": "BrOP5", + "elements": [ + { + "type": "rich_text_section", + "elements": [ + { + "type": "text", + "text": "Can you share the the job and events?\nAlso " + }, + { + "type": "user", + "user_id": "U02MK6YNAQ5" + } + ] + } + ] + } + ], + "team": "T01CWUYP5AR", + "thread_ts": "1694583867.900909", + "parent_user_id": "U05A1D80QKF" + }, + { + "type": "message", + "text": "Sure, sharing Job and events.", + "files": [ + { + "id": "F05S21H1LHK", + "created": 1694599423, + "timestamp": 1694599423, + "name": "spark_events.txt", + "title": "spark_events.txt", + "mimetype": "text/plain", + "filetype": "text", + "pretty_type": "Plain Text", + "user": "U05A1D80QKF", + "user_team": "T01CWUYP5AR", + "editable": true, + "size": 49881, + "mode": "snippet", + "is_external": false, + "external_type": "", + "is_public": true, + "public_url_shared": false, + "display_as_bot": false, + "username": "", + "url_private": "https://files.slack.com/files-pri/T01CWUYP5AR-F05S21H1LHK/spark_events.txt", + "url_private_download": "https://files.slack.com/files-pri/T01CWUYP5AR-F05S21H1LHK/download/spark_events.txt", + "permalink": "https://openlineage.slack.com/files/U05A1D80QKF/F05S21H1LHK/spark_events.txt", + "permalink_public": "https://slack-files.com/T01CWUYP5AR-F05S21H1LHK-d330f2f2ea", + "edit_link": "https://openlineage.slack.com/files/U05A1D80QKF/F05S21H1LHK/spark_events.txt/edit", + "preview": "{\"eventTime\": \"2023-09-12T20:44:10.764Z\", \"producer\": \"https://github.com/OpenLineage/OpenLineage/tree/1.1.0/integration/spark\", \"schemaURL\": \"https://openlineage.io/spec/2-0-2/OpenLineage.json#/$defs/RunEvent\", \"eventType\": \"START\", \"run\": {\"runId\": \"9293fb3d-bbe9-4237-b518-719a7c0f149d\", \"facets\": {\"spark.logicalPlan\": {\"_producer\": \"https://github.com/OpenLineage/OpenLineage/tree/1.1.0/integration/spark\", \"_schemaURL\": \"https://openlineage.io/spec/2-0-2/OpenLineage.json#/$defs/RunFacet\", \"plan\": [{\"class\": \"org.apache.spark.sql.execution.datasources.SaveIntoDataSourceCommand\", \"num-children\": 0, \"query\": [{\"class\": \"org.apache.spark.sql.catalyst.plans.logical.Project\", \"num-children\": 1, \"projectList\": [[{\"class\": \"org.apache.spark.sql.catalyst.expressions.AttributeReference\", \"num-children\": 0, \"name\": \"customer_id\", \"dataType\": \"integer\", \"nullable\": true, \"metadata\": {}, \"exprId\": {\"product-class\": \"org.apache.spark.sql.catalyst.expressions.ExprId\", \"id\": 497, \"jvmId\": \"bca387c7-9171-4d47-8061-7031cec5e...", + "preview_highlight": "
\n
\n
{"eventTime": "2023-09-12T20:44:10.764Z", "producer": "https://github.com/OpenLineage/OpenLineage/tree/1.1.0/integration/spark", "schemaURL": "https://openlineage.io/spec/2-0-2/OpenLineage.json#/$defs/RunEvent", "eventType": "START", "run": {"runId": "9293fb3d-bbe9-4237-b518-719a7c0f149d", "facets": {"spark.logicalPlan": {"_producer": "https://github.com/OpenLineage/OpenLineage/tree/1.1.0/integration/spark", "_schemaURL": "https://openlineage.io/spec/2-0-2/OpenLineage.json#/$defs/RunFacet", "plan": [{"class": "org.apache.spark.sql.execution.datasources.SaveIntoDataSourceCommand", "num-children": 0, "query": [{"class": "org.apache.spark.sql.catalyst.plans.logical.Project", "num-children": 1, "projectList": [[{"class": "org.apache.spark.sql.catalyst.expressions.AttributeReference", "num-children": 0, "name": "customer_id", "dataType": "integer", "nullable": true, "metadata": {}, "exprId": {"product-class": "org.apache.spark.sql.catalyst.expressions.ExprId", "id": 497, "jvmId": "bca387c7-9171-4d47-8061-7031cec5e...
\n
\n
\n", + "lines": 5, + "lines_more": 4, + "preview_is_truncated": true, + "is_starred": false, + "has_rich_preview": false, + "file_access": "visible" + }, + { + "id": "F05S4UHH4GJ", + "mode": "tombstone" + } + ], + "upload": false, + "user": "U05A1D80QKF", + "ts": "1694599429.520289", + "blocks": [ + { + "type": "rich_text", + "block_id": "Gmvxl", + "elements": [ + { + "type": "rich_text_section", + "elements": [ + { + "type": "text", + "text": "Sure, sharing Job and events." + } + ] + } + ] + } + ], + "client_msg_id": "6dbc15c5-b1f3-4cf5-8c9f-8e7a9d0e3582", + "thread_ts": "1694583867.900909", + "parent_user_id": "U05A1D80QKF" + }, + { + "type": "message", + "text": "", + "files": [ + { + "id": "F05S4S20MDZ", + "created": 1694599579, + "timestamp": 1694599579, + "name": "etl-mysql.py", + "title": "etl-mysql.py", + "mimetype": "text/plain", + "filetype": "python", + "pretty_type": "Python", + "user": "U05A1D80QKF", + "user_team": "T01CWUYP5AR", + "editable": true, + "size": 2005, + "mode": "snippet", + "is_external": false, + "external_type": "", + "is_public": true, + "public_url_shared": false, + "display_as_bot": false, + "username": "", + "url_private": "https://files.slack.com/files-pri/T01CWUYP5AR-F05S4S20MDZ/etl-mysql.py", + "url_private_download": "https://files.slack.com/files-pri/T01CWUYP5AR-F05S4S20MDZ/download/etl-mysql.py", + "permalink": "https://openlineage.slack.com/files/U05A1D80QKF/F05S4S20MDZ/etl-mysql.py", + "permalink_public": "https://slack-files.com/T01CWUYP5AR-F05S4S20MDZ-b2be55e90f", + "edit_link": "https://openlineage.slack.com/files/U05A1D80QKF/F05S4S20MDZ/etl-mysql.py/edit", + "preview": "from pyspark.sql import SparkSession\nfrom pyspark.sql.functions import col, concat_ws, year, month, dayofmonth\nfrom pyspark.sql.types import StructType, StructField, StringType, IntegerType\n\n", + "preview_highlight": "
\n
\n
from pyspark.sql import SparkSession
\n
from pyspark.sql.functions import col, concat_ws, year, month, dayofmonth
\n
from pyspark.sql.types import StructType, StructField, StringType, IntegerType
\n
\n
\n
\n", + "lines": 57, + "lines_more": 52, + "preview_is_truncated": true, + "is_starred": false, + "has_rich_preview": false, + "file_access": "visible" + } + ], + "upload": false, + "user": "U05A1D80QKF", + "display_as_bot": false, + "ts": "1694599581.950579", + "client_msg_id": "50568519-5364-4dff-9fbe-750dd6896192", + "thread_ts": "1694583867.900909", + "parent_user_id": "U05A1D80QKF" + }, + { + "client_msg_id": "de52d2d9-2547-40f3-b0c9-f1084212f451", + "type": "message", + "text": "Hi <@U05A1D80QKF>,\n\nThanks for providing such a detailed description of the problem.\n\nIt is not expected behaviour, it's an issue. The events correspond to the same logical plan which for some reason lead to sending two OL events. Is it reproducible aka. does it occur each time? If yes, we please feel free to raise an issue for that.\n\nWe have added in recent months several tests to verify amount of OL events being generated but we haven't tested it that way with JDBC. BTW. will the same happen if you write your data `df_transformed` to a file (like parquet file) ?", + "user": "U02MK6YNAQ5", + "ts": "1694601542.208509", + "blocks": [ + { + "type": "rich_text", + "block_id": "Q7/y4", + "elements": [ + { + "type": "rich_text_section", + "elements": [ + { + "type": "text", + "text": "Hi " + }, + { + "type": "user", + "user_id": "U05A1D80QKF" + }, + { + "type": "text", + "text": ",\n\nThanks for providing such a detailed description of the problem.\n\nIt is not expected behaviour, it's an issue. The events correspond to the same logical plan which for some reason lead to sending two OL events. Is it reproducible aka. does it occur each time? If yes, we please feel free to raise an issue for that.\n\nWe have added in recent months several tests to verify amount of OL events being generated but we haven't tested it that way with JDBC. BTW. will the same happen if you write your data " + }, + { + "type": "text", + "text": "df_transformed", + "style": { + "code": true + } + }, + { + "type": "text", + "text": " to a file (like parquet file) ?" + } + ] + } + ] + } + ], + "team": "T01CWUYP5AR", + "thread_ts": "1694583867.900909", + "parent_user_id": "U05A1D80QKF", + "reactions": [ + { + "name": "gratitude-thank-you", + "users": [ + "U05A1D80QKF" + ], + "count": 1 + } + ] + }, + { + "client_msg_id": "31906db3-adc9-461c-bcb5-89d606785db8", + "type": "message", + "text": "Thanks <@U02MK6YNAQ5>, will confirm about writing to file and get back.", + "user": "U05A1D80QKF", + "ts": "1694604483.876959", + "blocks": [ + { + "type": "rich_text", + "block_id": "OLVgV", + "elements": [ + { + "type": "rich_text_section", + "elements": [ + { + "type": "text", + "text": "Thanks " + }, + { + "type": "user", + "user_id": "U02MK6YNAQ5" + }, + { + "type": "text", + "text": ", will confirm about writing to file and get back." + } + ] + } + ] + } + ], + "team": "T01CWUYP5AR", + "thread_ts": "1694583867.900909", + "parent_user_id": "U05A1D80QKF" + }, + { + "client_msg_id": "c383dd11-9d69-4616-937f-31661d7f4c5d", + "type": "message", + "text": "And yes, the issue is reproducible. Will raise an issue for this.", + "user": "U05A1D80QKF", + "ts": "1694604815.950619", + "blocks": [ + { + "type": "rich_text", + "block_id": "lJ2OI", + "elements": [ + { + "type": "rich_text_section", + "elements": [ + { + "type": "text", + "text": "And yes, the issue is reproducible. Will raise an issue for this." + } + ] + } + ] + } + ], + "team": "T01CWUYP5AR", + "thread_ts": "1694583867.900909", + "parent_user_id": "U05A1D80QKF" + }, + { + "client_msg_id": "7571e602-ec66-4c82-b7e3-9f0b31e2ca91", + "type": "message", + "text": "even if you write onto a file?", + "user": "U02MK6YNAQ5", + "ts": "1694604834.201719", + "blocks": [ + { + "type": "rich_text", + "block_id": "1EGL5", + "elements": [ + { + "type": "rich_text_section", + "elements": [ + { + "type": "text", + "text": "even if you write onto a file?" + } + ] + } + ] + } + ], + "team": "T01CWUYP5AR", + "thread_ts": "1694583867.900909", + "parent_user_id": "U05A1D80QKF" + }, + { + "client_msg_id": "037e36e4-0098-4634-afe3-f505f16631f1", + "type": "message", + "text": "Yes, even when I write to a parquet file.", + "user": "U05A1D80QKF", + "ts": "1694605041.849429", + "blocks": [ + { + "type": "rich_text", + "block_id": "hyQSb", + "elements": [ + { + "type": "rich_text_section", + "elements": [ + { + "type": "text", + "text": "Yes, even when I write to a parquet file." + } + ] + } + ] + } + ], + "team": "T01CWUYP5AR", + "thread_ts": "1694583867.900909", + "parent_user_id": "U05A1D80QKF" + }, + { + "client_msg_id": "4cd3fe1c-1c46-40c1-abc8-5b44bef807ca", + "type": "message", + "text": "ok. i think i was able to reproduce it locally with ", + "user": "U02MK6YNAQ5", + "ts": "1694605768.036149", + "blocks": [ + { + "type": "rich_text", + "block_id": "ihnk0", + "elements": [ + { + "type": "rich_text_section", + "elements": [ + { + "type": "text", + "text": "ok. i think i was able to reproduce it locally with " + }, + { + "type": "link", + "url": "https://github.com/OpenLineage/OpenLineage/pull/2103/files" + } + ] + } + ] + } + ], + "team": "T01CWUYP5AR", + "thread_ts": "1694583867.900909", + "parent_user_id": "U05A1D80QKF" + }, + { + "client_msg_id": "741bf33c-bdbe-4bb1-b742-360dd6ac5675", + "type": "message", + "text": "Opened an issue: ", + "user": "U05A1D80QKF", + "ts": "1694606171.447199", + "blocks": [ + { + "type": "rich_text", + "block_id": "JNjNs", + "elements": [ + { + "type": "rich_text_section", + "elements": [ + { + "type": "text", + "text": "Opened an issue: " + }, + { + "type": "link", + "url": "https://github.com/OpenLineage/OpenLineage/issues/2104" + } + ] + } + ] + } + ], + "team": "T01CWUYP5AR", + "thread_ts": "1694583867.900909", + "parent_user_id": "U05A1D80QKF" + }, + { + "client_msg_id": "34debb23-2b95-4bf5-bf84-416dd485a755", + "type": "message", + "text": "<@U02MK6YNAQ5> I see that the is work in progress. Any rough estimate on when we can expect this fix to be released?", + "user": "U05A1D80QKF", + "ts": "1695673929.069669", + "blocks": [ + { + "type": "rich_text", + "block_id": "sgyzW", + "elements": [ + { + "type": "rich_text_section", + "elements": [ + { + "type": "user", + "user_id": "U02MK6YNAQ5" + }, + { + "type": "text", + "text": " I see that the " + }, + { + "type": "link", + "url": "https://github.com/OpenLineage/OpenLineage/pull/2103", + "text": "PR" + }, + { + "type": "text", + "text": " is work in progress. Any rough estimate on when we can expect this fix to be released?" + } + ] + } + ] + } + ], + "team": "T01CWUYP5AR", + "thread_ts": "1694583867.900909", + "parent_user_id": "U05A1D80QKF" + }, + { + "client_msg_id": "ec12c784-190b-4cae-a481-14543076ab1d", + "type": "message", + "text": "<@U05A1D80QKF> put a comment within your issue. it's a bug we need to solve but I cannot bring any estimates today.", + "user": "U02MK6YNAQ5", + "ts": "1695713523.494129", + "blocks": [ + { + "type": "rich_text", + "block_id": "LkN+M", + "elements": [ + { + "type": "rich_text_section", + "elements": [ + { + "type": "user", + "user_id": "U05A1D80QKF" + }, + { + "type": "text", + "text": " put a comment within your issue. it's a bug we need to solve but I cannot bring any estimates today." + } + ] + } + ] + } + ], + "team": "T01CWUYP5AR", + "thread_ts": "1694583867.900909", + "parent_user_id": "U05A1D80QKF" + }, + { + "client_msg_id": "442caa17-2deb-43ee-abe0-7de40a38f803", + "type": "message", + "text": "Thanks for update <@U02MK6YNAQ5>, also please look into comment. It might related and I'm not sure if expected behaviour.", + "user": "U05A1D80QKF", + "ts": "1695717183.779589", + "blocks": [ + { + "type": "rich_text", + "block_id": "smZz1", + "elements": [ + { + "type": "rich_text_section", + "elements": [ + { + "type": "text", + "text": "Thanks for update " + }, + { + "type": "user", + "user_id": "U02MK6YNAQ5" + }, + { + "type": "text", + "text": ", also please look into " + }, + { + "type": "link", + "url": "https://github.com/OpenLineage/OpenLineage/issues/2104#issuecomment-1735065087", + "text": "this" + }, + { + "type": "text", + "text": " comment. It might related and I'm not sure if expected behaviour." + } + ] + } + ] + } + ], + "team": "T01CWUYP5AR", + "attachments": [ + { + "id": 1, + "footer_icon": "https://slack.github.com/static/img/favicon-neutral.png", + "ts": 1695716708, + "color": "24292f", + "bot_id": "B01VA0FB340", + "app_unfurl_url": "https://github.com/OpenLineage/OpenLineage/issues/2104#issuecomment-1735065087", + "is_app_unfurl": true, + "app_id": "A01BP7R4KNY", + "fallback": "Comment on #2104 [Spark Integration] Receiving duplicate OpenLineage events for a Spark Job", + "text": "Hi I understand your point. I've also come across cases where the duplicate events have the exact same payload with no difference in facets. I'm assuming there should be some diff in the payload for the backend to merge them. I've also seen a single Job emit 7 events and there is no fixed pattern for this. \nI also agree that there should be only a single START and a single COMPLETE event for any Job since this what the OL spec says.\n\nThis Job emits 7 events (5 Start and 2 Complete):\n\n```\nfrom pyspark.sql import SparkSession\nfrom pyspark.sql.functions import col\nfrom pyspark.sql.functions import lit\nimport random\n\nspark = (SparkSession.builder.master('local')\n .appName('spark-pipeline-v1')\n .config('spark.jars.packages', \"io.openlineage:openlineage-spark:1.1.0,\"\n \"mysql:mysql-connector-java:8.0.33,\"\n \"net.snowflake:snowflake-jdbc:3.13.14,\"\n \"net.snowflake:spark-snowflake_2.12:2.10.0-spark_3.2\")\n .config('spark.extraListeners', 'io.openlineage.spark.agent.OpenLineageSparkListener')\n .config('spark.openlineage.transport.type', 'http')\n .config('spark.openlineage.transport.url', '')\n .config('spark.openlineage.transport.endpoint', '/events/openlineage/spark/api/v1/lineage')\n .config('spark.openlineage.namespace', 'staging')\n .config('spark.openlineage.transport.auth.type', 'api_key')\n .config('spark.openlineage.transport.auth.apiKey', 'test-key')\n .config('spark.openlineage.parentJobName', 'suraj-test-job')\n .config('spark.openlineage.parentRunId', 'acd-eheh-ththth-wnjwnj')\n .getOrCreate())\n\n\nmysql_connection_properties = {\n \"user\": \"\",\n \"password\": \"\",\n \"driver\": \"com.mysql.cj.jdbc.Driver\",\n}\n\nsnowflake_options = {\n \"sfURL\": \".\",\n \"sfUser\": \"\",\n \"sfPassword\": \"\",\n \"sfDatabase\": \"ANALYTICS\",\n \"sfWarehouse\": \"COMPUTE_WH\",\n \"sfSchema\": \"PUBLIC\",\n \"sfRole\": \"ACCOUNTADMIN\",\n}\n\nmysql_url = \"\"\n\ncats_df = spark.read.jdbc(url=mysql_url, table=\"cats\", properties=mysql_connection_properties)\ndogs_df = spark.read.jdbc(url=mysql_url, table=\"dogs\", properties=mysql_connection_properties)\n\ncats_df = cats_df.withColumnRenamed(\"Country\", \"cat_name\")\n\njoined_df = cats_df.join(dogs_df, on=\"owner\", how=\"inner\")\n\nfiltered_df = joined_df.filter(col(\"cat_name\") == 'Cookie')\n\nagg_df = filtered_df.groupBy(\"cat_name\").count()\n\nagg_df = agg_df.withColumn(\"id\", lit(random.randint(0, 1000)))\n\nagg_df.write \\\n .format(\"snowflake\") \\\n .options(**snowflake_options) \\\n .option(\"dbtable\", \"COUNT_TABLE\") \\\n .mode(\"overwrite\") \\\n .save()\n\nspark.stop()\n```", + "title": "Comment on #2104 [Spark Integration] Receiving duplicate OpenLineage events for a Spark Job", + "title_link": "https://github.com/OpenLineage/OpenLineage/issues/2104#issuecomment-1735065087", + "footer": "", + "mrkdwn_in": [ + "text" + ] + } + ], + "thread_ts": "1694583867.900909", + "parent_user_id": "U05A1D80QKF" + } + ] + }, + { + "client_msg_id": "45c0b47a-699b-47d1-bc76-b4028a7d2d13", + "type": "message", + "text": " has anyone succeded in getting a custom extractor to work in GCP Cloud Composer or AWS MWAA, seems like there is no way", + "user": "U05QL7LN2GH", + "ts": "1694553961.188719", + "blocks": [ + { + "type": "rich_text", + "block_id": "og69G", + "elements": [ + { + "type": "rich_text_section", + "elements": [ + { + "type": "broadcast", + "range": "here" + }, + { + "type": "text", + "text": " has anyone succeded in getting a custom extractor to work in GCP Cloud Composer or AWS MWAA, seems like there is no way" + } + ] + } + ] + } + ], + "team": "T01CWUYP5AR", + "thread_ts": "1694553961.188719", + "reply_count": 1, + "reply_users_count": 1, + "latest_reply": "1694554469.757559", + "reply_users": [ + "U01HVNU6A4C" + ], + "is_locked": false, + "subscribed": false, + "replies": [ + { + "client_msg_id": "2cbfffbe-1026-4f2e-a8a7-8091b79301aa", + "type": "message", + "text": "I'm getting quite close with MWAA. See .", + "user": "U01HVNU6A4C", + "ts": "1694554469.757559", + "blocks": [ + { + "type": "rich_text", + "block_id": "cYLI+", + "elements": [ + { + "type": "rich_text_section", + "elements": [ + { + "type": "text", + "text": "I'm getting quite close with MWAA. See " + }, + { + "type": "link", + "url": "https://openlineage.slack.com/archives/C01CK9T7HKR/p1692743745585879" + }, + { + "type": "text", + "text": "." + } + ] + } + ] + } + ], + "team": "T01CWUYP5AR", + "edited": { + "user": "U01HVNU6A4C", + "ts": "1694554486.000000" + }, + "attachments": [ + { + "from_url": "https://openlineage.slack.com/archives/C01CK9T7HKR/p1692743745585879", + "ts": "1692743745.585879", + "author_id": "U01HVNU6A4C", + "channel_id": "C01CK9T7HKR", + "channel_team": "T01CWUYP5AR", + "is_msg_unfurl": true, + "is_thread_root_unfurl": true, + "message_blocks": [ + { + "team": "T01CWUYP5AR", + "channel": "C01CK9T7HKR", + "ts": "1692743745.585879", + "message": { + "blocks": [ + { + "type": "rich_text", + "block_id": "swU0", + "elements": [ + { + "type": "rich_text_section", + "elements": [ + { + "type": "text", + "text": "Has anyone managed to get the OL Airflow integration to work on AWS MWAA? We've tried pretty much every trick but still ended up with the following error:\n" + } + ] + }, + { + "type": "rich_text_preformatted", + "elements": [ + { + "type": "text", + "text": "Broken plugin: [openlineage.airflow.plugin] No module named 'openlineage.airflow'; 'openlineage' is not a package" + } + ], + "border": 0 + }, + { + "type": "rich_text_section", + "elements": [] + } + ] + } + ] + } + } + ], + "id": 1, + "original_url": "https://openlineage.slack.com/archives/C01CK9T7HKR/p1692743745585879", + "fallback": "[August 22nd, 2023 3:35 PM] mars: Has anyone managed to get the OL Airflow integration to work on AWS MWAA? We've tried pretty much every trick but still ended up with the following error:\n```Broken plugin: [openlineage.airflow.plugin] No module named 'openlineage.airflow'; 'openlineage' is not a package```", + "text": "Has anyone managed to get the OL Airflow integration to work on AWS MWAA? We've tried pretty much every trick but still ended up with the following error:\n```Broken plugin: [openlineage.airflow.plugin] No module named 'openlineage.airflow'; 'openlineage' is not a package```", + "author_name": "Mars Lan", + "author_link": "https://openlineage.slack.com/team/U01HVNU6A4C", + "author_icon": "https://avatars.slack-edge.com/2020-12-28/1598525158821_e70f0b60ad8c72e52398_48.jpg", + "author_subname": "Mars Lan", + "mrkdwn_in": [ + "text" + ], + "footer": "Thread in Slack Conversation" + } + ], + "thread_ts": "1694553961.188719", + "parent_user_id": "U05QL7LN2GH" + } + ] + }, + { + "type": "message", + "text": "I am trying to run Google Cloud Composer where i have added the openlineage-airflow pypi packagae as a dependency and have added the env OPENLINEAGE_EXTRACTORS to point to my custom extractor. I have added a folder by name dependencies and inside that i have placed my extractor file, and the path given to OPENLINEAGE_EXTRACTORS is dependencies.<file_name>.<extractor_class_name>…still it fails with the exception saying No module named ‘dependencies’. Can anyone kindly help me out on correcting my mistake", + "files": [ + { + "id": "F05RM6EV6DV", + "created": 1694545739, + "timestamp": 1694545739, + "name": "Screenshot 2023-09-13 at 12.38.55 AM.png", + "title": "Screenshot 2023-09-13 at 12.38.55 AM.png", + "mimetype": "image/png", + "filetype": "png", + "pretty_type": "PNG", + "user": "U05QL7LN2GH", + "user_team": "T01CWUYP5AR", + "editable": false, + "size": 951188, + "mode": "hosted", + "is_external": false, + "external_type": "", + "is_public": true, + "public_url_shared": false, + "display_as_bot": false, + "username": "", + "url_private": "https://files.slack.com/files-pri/T01CWUYP5AR-F05RM6EV6DV/screenshot_2023-09-13_at_12.38.55_am.png", + "url_private_download": "https://files.slack.com/files-pri/T01CWUYP5AR-F05RM6EV6DV/download/screenshot_2023-09-13_at_12.38.55_am.png", + "media_display_type": "unknown", + "thumb_64": "https://files.slack.com/files-tmb/T01CWUYP5AR-F05RM6EV6DV-e20cfb50c7/screenshot_2023-09-13_at_12.38.55_am_64.png", + "thumb_80": "https://files.slack.com/files-tmb/T01CWUYP5AR-F05RM6EV6DV-e20cfb50c7/screenshot_2023-09-13_at_12.38.55_am_80.png", + "thumb_360": "https://files.slack.com/files-tmb/T01CWUYP5AR-F05RM6EV6DV-e20cfb50c7/screenshot_2023-09-13_at_12.38.55_am_360.png", + "thumb_360_w": 360, + "thumb_360_h": 122, + "thumb_480": "https://files.slack.com/files-tmb/T01CWUYP5AR-F05RM6EV6DV-e20cfb50c7/screenshot_2023-09-13_at_12.38.55_am_480.png", + "thumb_480_w": 480, + "thumb_480_h": 162, + "thumb_160": "https://files.slack.com/files-tmb/T01CWUYP5AR-F05RM6EV6DV-e20cfb50c7/screenshot_2023-09-13_at_12.38.55_am_160.png", + "thumb_720": "https://files.slack.com/files-tmb/T01CWUYP5AR-F05RM6EV6DV-e20cfb50c7/screenshot_2023-09-13_at_12.38.55_am_720.png", + "thumb_720_w": 720, + "thumb_720_h": 243, + "thumb_800": "https://files.slack.com/files-tmb/T01CWUYP5AR-F05RM6EV6DV-e20cfb50c7/screenshot_2023-09-13_at_12.38.55_am_800.png", + "thumb_800_w": 800, + "thumb_800_h": 270, + "thumb_960": "https://files.slack.com/files-tmb/T01CWUYP5AR-F05RM6EV6DV-e20cfb50c7/screenshot_2023-09-13_at_12.38.55_am_960.png", + "thumb_960_w": 960, + "thumb_960_h": 324, + "thumb_1024": "https://files.slack.com/files-tmb/T01CWUYP5AR-F05RM6EV6DV-e20cfb50c7/screenshot_2023-09-13_at_12.38.55_am_1024.png", + "thumb_1024_w": 1024, + "thumb_1024_h": 346, + "original_w": 3024, + "original_h": 1022, + "thumb_tiny": "AwAQADDRx2pefWg5zRQAtFJ+FHNAC0YpMmjmgD//2Q==", + "permalink": "https://openlineage.slack.com/files/U05QL7LN2GH/F05RM6EV6DV/screenshot_2023-09-13_at_12.38.55_am.png", + "permalink_public": "https://slack-files.com/T01CWUYP5AR-F05RM6EV6DV-62656c8fb4", + "is_starred": false, + "has_rich_preview": false, + "file_access": "visible" + } + ], + "upload": false, + "user": "U05QL7LN2GH", + "display_as_bot": false, + "ts": "1694545905.974339", + "blocks": [ + { + "type": "rich_text", + "block_id": "M37Ut", + "elements": [ + { + "type": "rich_text_section", + "elements": [ + { + "type": "text", + "text": "I am trying to run Google Cloud Composer where i have added the openlineage-airflow pypi packagae as a dependency and have added the env OPENLINEAGE_EXTRACTORS to point to my custom extractor. I have added a folder by name dependencies and inside that i have placed my extractor file, and the path given to OPENLINEAGE_EXTRACTORS is dependencies..…still it fails with the exception saying No module named ‘dependencies’. Can anyone kindly help me out on correcting my mistake" + } + ] + } + ] + } + ], + "client_msg_id": "96b4074c-8175-4e0c-a4d2-480c8ad53912", + "thread_ts": "1694545905.974339", + "reply_count": 56, + "reply_users_count": 4, + "latest_reply": "1694849569.693149", + "reply_users": [ + "U01HNKK4XAM", + "U05QL7LN2GH", + "U01RA9B5GG2", + "U02S6F54MAB" + ], + "is_locked": false, + "subscribed": false, + "replies": [ + { + "client_msg_id": "6699d029-37b4-46a5-a617-a3596fa43622", + "type": "message", + "text": "Hey <@U05QL7LN2GH>, can you share some details on which versions of airflow and openlineage you’re using?", + "user": "U01HNKK4XAM", + "ts": "1694553336.948439", + "blocks": [ + { + "type": "rich_text", + "block_id": "3T9v9", + "elements": [ + { + "type": "rich_text_section", + "elements": [ + { + "type": "text", + "text": "Hey " + }, + { + "type": "user", + "user_id": "U05QL7LN2GH" + }, + { + "type": "text", + "text": ", can you share some details on which versions of airflow and openlineage you’re using?" + } + ] + } + ] + } + ], + "team": "T01CWUYP5AR", + "thread_ts": "1694545905.974339", + "parent_user_id": "U05QL7LN2GH" + }, + { + "client_msg_id": "2983444d-e79f-47e1-b84f-52c510893b14", + "type": "message", + "text": "airflow ---> 2.5.3, openlinegae-airflow ---> 1.1.0", + "user": "U05QL7LN2GH", + "ts": "1694553386.763359", + "blocks": [ + { + "type": "rich_text", + "block_id": "zROwd", + "elements": [ + { + "type": "rich_text_section", + "elements": [ + { + "type": "text", + "text": "airflow ---> 2.5.3, openlinegae-airflow ---> 1.1.0" + } + ] + } + ] + } + ], + "team": "T01CWUYP5AR", + "thread_ts": "1694545905.974339", + "parent_user_id": "U05QL7LN2GH" + }, + { + "client_msg_id": "abe58931-c3fd-4e56-87de-4c9f035cfdbe", + "type": "message", + "text": "```import traceback\nimport uuid\nfrom typing import List, Optional\n\nfrom openlineage.airflow.extractors.base import BaseExtractor, TaskMetadata\nfrom openlineage.airflow.utils import get_job_name\n\n\nclass BigQueryInsertJobExtractor(BaseExtractor):\n def __init__(self, operator):\n super().__init__(operator)\n\n @classmethod\n def get_operator_classnames(cls) -> List[str]:\n return ['BigQueryInsertJobOperator']\n\n def extract(self) -> Optional[TaskMetadata]:\n return None\n\n def extract_on_complete(self, task_instance) -> Optional[TaskMetadata]:\n self.log.debug(f\"JEEVAN ---> extract_on_complete({task_instance})\")\n random_uuid = str(uuid.uuid4())\n self.log.debug(f\"JEEVAN ---> Randomly Generated UUID --> {random_uuid}\")\n\n self.operator.job_id = random_uuid\n\n return TaskMetadata(\n name=get_job_name(task=self.operator)\n )```", + "user": "U05QL7LN2GH", + "ts": "1694555108.611609", + "blocks": [ + { + "type": "rich_text", + "block_id": "yJdTD", + "elements": [ + { + "type": "rich_text_preformatted", + "elements": [ + { + "type": "text", + "text": "import traceback\nimport uuid\nfrom typing import List, Optional\n\nfrom openlineage.airflow.extractors.base import BaseExtractor, TaskMetadata\nfrom openlineage.airflow.utils import get_job_name\n\n\nclass BigQueryInsertJobExtractor(BaseExtractor):\n def __init__(self, operator):\n super().__init__(operator)\n\n @classmethod\n def get_operator_classnames(cls) -> List[str]:\n return ['BigQueryInsertJobOperator']\n\n def extract(self) -> Optional[TaskMetadata]:\n return None\n\n def extract_on_complete(self, task_instance) -> Optional[TaskMetadata]:\n self.log.debug(f\"JEEVAN ---> extract_on_complete({task_instance})\")\n random_uuid = str(uuid.uuid4())\n self.log.debug(f\"JEEVAN ---> Randomly Generated UUID --> {random_uuid}\")\n\n self.operator.job_id = random_uuid\n\n return TaskMetadata(\n name=get_job_name(task=self.operator)\n )" + } + ], + "border": 0 + } + ] + } + ], + "team": "T01CWUYP5AR", + "thread_ts": "1694545905.974339", + "parent_user_id": "U05QL7LN2GH" + }, + { + "client_msg_id": "e3536e9c-8378-41f5-b2d4-9cc9914646c8", + "type": "message", + "text": "this is the custom extractor code that im trying with", + "user": "U05QL7LN2GH", + "ts": "1694555124.052409", + "blocks": [ + { + "type": "rich_text", + "block_id": "+dZNQ", + "elements": [ + { + "type": "rich_text_section", + "elements": [ + { + "type": "text", + "text": "this is the custom extractor code that im trying with" + } + ] + } + ] + } + ], + "team": "T01CWUYP5AR", + "thread_ts": "1694545905.974339", + "parent_user_id": "U05QL7LN2GH" + }, + { + "client_msg_id": "56fb30af-3e45-469f-a4d7-f5f377ae8fa9", + "type": "message", + "text": "thanks <@U05QL7LN2GH>, will try to take a deeper look tomorrow", + "user": "U01HNKK4XAM", + "ts": "1694567402.289539", + "blocks": [ + { + "type": "rich_text", + "block_id": "chu5h", + "elements": [ + { + "type": "rich_text_section", + "elements": [ + { + "type": "text", + "text": "thanks " + }, + { + "type": "user", + "user_id": "U05QL7LN2GH" + }, + { + "type": "text", + "text": ", will try to take a deeper look tomorrow" + } + ] + } + ] + } + ], + "team": "T01CWUYP5AR", + "thread_ts": "1694545905.974339", + "parent_user_id": "U05QL7LN2GH" + }, + { + "client_msg_id": "2dc50680-0b06-466b-8b43-f4faec6fe389", + "type": "message", + "text": "`No module named 'dependencies'.`\nThis sounds like general Python problem", + "user": "U01RA9B5GG2", + "ts": "1694606066.296109", + "blocks": [ + { + "type": "rich_text", + "block_id": "jSK0Q", + "elements": [ + { + "type": "rich_text_section", + "elements": [ + { + "type": "text", + "text": "No module named 'dependencies'.", + "style": { + "code": true + } + }, + { + "type": "text", + "text": "\nThis sounds like general Python problem" + } + ] + } + ] + } + ], + "team": "T01CWUYP5AR", + "thread_ts": "1694545905.974339", + "parent_user_id": "U05QL7LN2GH" + }, + { + "client_msg_id": "488b814e-8149-44fa-a4eb-47578a90fc2d", + "type": "message", + "text": "", + "user": "U01RA9B5GG2", + "ts": "1694606112.308969", + "blocks": [ + { + "type": "rich_text", + "block_id": "wNpXH", + "elements": [ + { + "type": "rich_text_section", + "elements": [ + { + "type": "link", + "url": "https://stackoverflow.com/questions/69991553/how-to-import-custom-modules-in-cloud-composer" + } + ] + } + ] + } + ], + "team": "T01CWUYP5AR", + "attachments": [ + { + "from_url": "https://stackoverflow.com/questions/69991553/how-to-import-custom-modules-in-cloud-composer", + "thumb_url": "https://cdn.sstatic.net/Sites/stackoverflow/Img/apple-touch-icon@2.png?v=73d79a89bded", + "thumb_width": 316, + "thumb_height": 316, + "service_icon": "https://cdn.sstatic.net/Sites/stackoverflow/Img/apple-touch-icon.png?v=c78bd457575a", + "id": 1, + "original_url": "https://stackoverflow.com/questions/69991553/how-to-import-custom-modules-in-cloud-composer", + "fallback": "Stack Overflow: how to import custom modules in Cloud Composer", + "text": "I created a local project with apache Airflow and i want to run it in cloud composer. My project contains custom modules and a main file that calls them.\nExample : from src.kuzzle import KuzzleQuery", + "title": "how to import custom modules in Cloud Composer", + "title_link": "https://stackoverflow.com/questions/69991553/how-to-import-custom-modules-in-cloud-composer", + "service_name": "Stack Overflow" + } + ], + "thread_ts": "1694545905.974339", + "parent_user_id": "U05QL7LN2GH" + }, + { + "client_msg_id": "38a47c30-f268-4ad9-9462-97970aa94cb7", + "type": "message", + "text": "basically, if you're able to import the file from your dag code, OL should be able too", + "user": "U01RA9B5GG2", + "ts": "1694606188.664139", + "blocks": [ + { + "type": "rich_text", + "block_id": "NmblC", + "elements": [ + { + "type": "rich_text_section", + "elements": [ + { + "type": "text", + "text": "basically, if you're able to import the file from your dag code, OL should be able too" + } + ] + } + ] + } + ], + "team": "T01CWUYP5AR", + "thread_ts": "1694545905.974339", + "parent_user_id": "U05QL7LN2GH" + }, + { + "client_msg_id": "61ec8716-1531-4611-be70-5520e7f8c613", + "type": "message", + "text": "The Problem is in the GCS Composer there is a component called Triggerer, which they say is used for deferrable operators…i have logged into that pod and i could see that the GCS Bucket is not mounted on this, but i am unable to understand why is the initialisation happening inside the triggerer pod", + "user": "U05QL7LN2GH", + "ts": "1694606472.982729", + "blocks": [ + { + "type": "rich_text", + "block_id": "8kk5D", + "elements": [ + { + "type": "rich_text_section", + "elements": [ + { + "type": "text", + "text": "The Problem is in the GCS Composer there is a component called Triggerer, which they say is used for deferrable operators…i have logged into that pod and i could see that the GCS Bucket is not mounted on this, but i am unable to understand why is the initialisation happening inside the triggerer pod" + } + ] + } + ] + } + ], + "team": "T01CWUYP5AR", + "thread_ts": "1694545905.974339", + "parent_user_id": "U05QL7LN2GH" + }, + { + "type": "message", + "text": "", + "files": [ + { + "id": "F05SUDUQEDN", + "created": 1694606486, + "timestamp": 1694606486, + "name": "Screenshot 2023-09-13 at 5.31.22 PM.png", + "title": "Screenshot 2023-09-13 at 5.31.22 PM.png", + "mimetype": "image/png", + "filetype": "png", + "pretty_type": "PNG", + "user": "U05QL7LN2GH", + "user_team": "T01CWUYP5AR", + "editable": false, + "size": 1542818, + "mode": "hosted", + "is_external": false, + "external_type": "", + "is_public": true, + "public_url_shared": false, + "display_as_bot": false, + "username": "", + "url_private": "https://files.slack.com/files-pri/T01CWUYP5AR-F05SUDUQEDN/screenshot_2023-09-13_at_5.31.22_pm.png", + "url_private_download": "https://files.slack.com/files-pri/T01CWUYP5AR-F05SUDUQEDN/download/screenshot_2023-09-13_at_5.31.22_pm.png", + "media_display_type": "unknown", + "thumb_64": "https://files.slack.com/files-tmb/T01CWUYP5AR-F05SUDUQEDN-b8a4896bfd/screenshot_2023-09-13_at_5.31.22_pm_64.png", + "thumb_80": "https://files.slack.com/files-tmb/T01CWUYP5AR-F05SUDUQEDN-b8a4896bfd/screenshot_2023-09-13_at_5.31.22_pm_80.png", + "thumb_360": "https://files.slack.com/files-tmb/T01CWUYP5AR-F05SUDUQEDN-b8a4896bfd/screenshot_2023-09-13_at_5.31.22_pm_360.png", + "thumb_360_w": 360, + "thumb_360_h": 161, + "thumb_480": "https://files.slack.com/files-tmb/T01CWUYP5AR-F05SUDUQEDN-b8a4896bfd/screenshot_2023-09-13_at_5.31.22_pm_480.png", + "thumb_480_w": 480, + "thumb_480_h": 214, + "thumb_160": "https://files.slack.com/files-tmb/T01CWUYP5AR-F05SUDUQEDN-b8a4896bfd/screenshot_2023-09-13_at_5.31.22_pm_160.png", + "thumb_720": "https://files.slack.com/files-tmb/T01CWUYP5AR-F05SUDUQEDN-b8a4896bfd/screenshot_2023-09-13_at_5.31.22_pm_720.png", + "thumb_720_w": 720, + "thumb_720_h": 321, + "thumb_800": "https://files.slack.com/files-tmb/T01CWUYP5AR-F05SUDUQEDN-b8a4896bfd/screenshot_2023-09-13_at_5.31.22_pm_800.png", + "thumb_800_w": 800, + "thumb_800_h": 357, + "thumb_960": "https://files.slack.com/files-tmb/T01CWUYP5AR-F05SUDUQEDN-b8a4896bfd/screenshot_2023-09-13_at_5.31.22_pm_960.png", + "thumb_960_w": 960, + "thumb_960_h": 428, + "thumb_1024": "https://files.slack.com/files-tmb/T01CWUYP5AR-F05SUDUQEDN-b8a4896bfd/screenshot_2023-09-13_at_5.31.22_pm_1024.png", + "thumb_1024_w": 1024, + "thumb_1024_h": 457, + "original_w": 3010, + "original_h": 1342, + "thumb_tiny": "AwAVADDS74xQRR/FQaAEpaTtS0AFFLRQAY5pGOKWmv2oATOaXNNHWnUxC5ozSUUDP//Z", + "permalink": "https://openlineage.slack.com/files/U05QL7LN2GH/F05SUDUQEDN/screenshot_2023-09-13_at_5.31.22_pm.png", + "permalink_public": "https://slack-files.com/T01CWUYP5AR-F05SUDUQEDN-edf19dc4d6", + "is_starred": false, + "has_rich_preview": false, + "file_access": "visible" + } + ], + "upload": false, + "user": "U05QL7LN2GH", + "display_as_bot": false, + "ts": "1694606492.739709", + "client_msg_id": "d19589eb-ea2b-4503-851a-1bff3ab03683", + "thread_ts": "1694545905.974339", + "parent_user_id": "U05QL7LN2GH" + }, + { + "client_msg_id": "63649a04-b4df-4ffe-ad4f-bc6857c3de86", + "type": "message", + "text": "> The Problem is in the GCS Composer there is a component called Triggerer, which they say is used for deferrable operators…i have logged into that pod and i could see that the GCS Bucket is not mounted on this, but i am unable to understand why is the initialisation happening inside the triggerer pod\nOL integration is not running on triggerer, only on worker and scheduler pods", + "user": "U01RA9B5GG2", + "ts": "1694606507.351309", + "blocks": [ + { + "type": "rich_text", + "block_id": "Dd/xA", + "elements": [ + { + "type": "rich_text_quote", + "elements": [ + { + "type": "text", + "text": "The Problem is in the GCS Composer there is a component called Triggerer, which they say is used for deferrable operators…i have logged into that pod and i could see that the GCS Bucket is not mounted on this, but i am unable to understand why is the initialisation happening inside the triggerer pod" + } + ] + }, + { + "type": "rich_text_section", + "elements": [ + { + "type": "text", + "text": "\nOL integration is not running on triggerer, only on worker and scheduler pods" + } + ] + } + ] + } + ], + "team": "T01CWUYP5AR", + "thread_ts": "1694545905.974339", + "parent_user_id": "U05QL7LN2GH" + }, + { + "type": "message", + "text": "", + "files": [ + { + "id": "F05SJ3DJ5CH", + "created": 1694606507, + "timestamp": 1694606507, + "name": "Screenshot 2023-09-13 at 5.31.44 PM.png", + "title": "Screenshot 2023-09-13 at 5.31.44 PM.png", + "mimetype": "image/png", + "filetype": "png", + "pretty_type": "PNG", + "user": "U05QL7LN2GH", + "user_team": "T01CWUYP5AR", + "editable": false, + "size": 331156, + "mode": "hosted", + "is_external": false, + "external_type": "", + "is_public": true, + "public_url_shared": false, + "display_as_bot": false, + "username": "", + "url_private": "https://files.slack.com/files-pri/T01CWUYP5AR-F05SJ3DJ5CH/screenshot_2023-09-13_at_5.31.44_pm.png", + "url_private_download": "https://files.slack.com/files-pri/T01CWUYP5AR-F05SJ3DJ5CH/download/screenshot_2023-09-13_at_5.31.44_pm.png", + "media_display_type": "unknown", + "thumb_64": "https://files.slack.com/files-tmb/T01CWUYP5AR-F05SJ3DJ5CH-d898f6fdcc/screenshot_2023-09-13_at_5.31.44_pm_64.png", + "thumb_80": "https://files.slack.com/files-tmb/T01CWUYP5AR-F05SJ3DJ5CH-d898f6fdcc/screenshot_2023-09-13_at_5.31.44_pm_80.png", + "thumb_360": "https://files.slack.com/files-tmb/T01CWUYP5AR-F05SJ3DJ5CH-d898f6fdcc/screenshot_2023-09-13_at_5.31.44_pm_360.png", + "thumb_360_w": 360, + "thumb_360_h": 125, + "thumb_480": "https://files.slack.com/files-tmb/T01CWUYP5AR-F05SJ3DJ5CH-d898f6fdcc/screenshot_2023-09-13_at_5.31.44_pm_480.png", + "thumb_480_w": 480, + "thumb_480_h": 167, + "thumb_160": "https://files.slack.com/files-tmb/T01CWUYP5AR-F05SJ3DJ5CH-d898f6fdcc/screenshot_2023-09-13_at_5.31.44_pm_160.png", + "thumb_720": "https://files.slack.com/files-tmb/T01CWUYP5AR-F05SJ3DJ5CH-d898f6fdcc/screenshot_2023-09-13_at_5.31.44_pm_720.png", + "thumb_720_w": 720, + "thumb_720_h": 251, + "thumb_800": "https://files.slack.com/files-tmb/T01CWUYP5AR-F05SJ3DJ5CH-d898f6fdcc/screenshot_2023-09-13_at_5.31.44_pm_800.png", + "thumb_800_w": 800, + "thumb_800_h": 279, + "thumb_960": "https://files.slack.com/files-tmb/T01CWUYP5AR-F05SJ3DJ5CH-d898f6fdcc/screenshot_2023-09-13_at_5.31.44_pm_960.png", + "thumb_960_w": 960, + "thumb_960_h": 335, + "thumb_1024": "https://files.slack.com/files-tmb/T01CWUYP5AR-F05SJ3DJ5CH-d898f6fdcc/screenshot_2023-09-13_at_5.31.44_pm_1024.png", + "thumb_1024_w": 1024, + "thumb_1024_h": 357, + "original_w": 3024, + "original_h": 1054, + "thumb_tiny": "AwAQADCrub1P50hZx/EfzNIaQ0ALvb+835mje395vzNNooAdvf8AvN+dJvf+8350lFAH/9k=", + "permalink": "https://openlineage.slack.com/files/U05QL7LN2GH/F05SJ3DJ5CH/screenshot_2023-09-13_at_5.31.44_pm.png", + "permalink_public": "https://slack-files.com/T01CWUYP5AR-F05SJ3DJ5CH-c6f8f73a8d", + "is_starred": false, + "has_rich_preview": false, + "file_access": "visible" + } + ], + "upload": false, + "user": "U05QL7LN2GH", + "display_as_bot": false, + "ts": "1694606513.120669", + "client_msg_id": "81c34c2f-282f-4b5a-9bc1-698ff96254ee", + "thread_ts": "1694545905.974339", + "parent_user_id": "U05QL7LN2GH" + }, + { + "client_msg_id": "396e095e-dded-49d2-8aaa-b0845d67f992", + "type": "message", + "text": "As you can see in this screenshot i am seeing the logs of the triggerer and it says clearly unable to import plugin openlineage", + "user": "U05QL7LN2GH", + "ts": "1694606606.171889", + "blocks": [ + { + "type": "rich_text", + "block_id": "vBkL+", + "elements": [ + { + "type": "rich_text_section", + "elements": [ + { + "type": "text", + "text": "As you can see in this screenshot i am seeing the logs of the triggerer and it says clearly unable to import plugin openlineage" + } + ] + } + ] + } + ], + "team": "T01CWUYP5AR", + "thread_ts": "1694545905.974339", + "parent_user_id": "U05QL7LN2GH" + }, + { + "type": "message", + "text": "", + "files": [ + { + "id": "F05SUDUQEDN", + "created": 1694606486, + "timestamp": 1694606486, + "name": "Screenshot 2023-09-13 at 5.31.22 PM.png", + "title": "Screenshot 2023-09-13 at 5.31.22 PM.png", + "mimetype": "image/png", + "filetype": "png", + "pretty_type": "PNG", + "user": "U05QL7LN2GH", + "user_team": "T01CWUYP5AR", + "editable": false, + "size": 1542818, + "mode": "hosted", + "is_external": false, + "external_type": "", + "is_public": true, + "public_url_shared": false, + "display_as_bot": false, + "username": "", + "url_private": "https://files.slack.com/files-pri/T01CWUYP5AR-F05SUDUQEDN/screenshot_2023-09-13_at_5.31.22_pm.png", + "url_private_download": "https://files.slack.com/files-pri/T01CWUYP5AR-F05SUDUQEDN/download/screenshot_2023-09-13_at_5.31.22_pm.png", + "media_display_type": "unknown", + "thumb_64": "https://files.slack.com/files-tmb/T01CWUYP5AR-F05SUDUQEDN-b8a4896bfd/screenshot_2023-09-13_at_5.31.22_pm_64.png", + "thumb_80": "https://files.slack.com/files-tmb/T01CWUYP5AR-F05SUDUQEDN-b8a4896bfd/screenshot_2023-09-13_at_5.31.22_pm_80.png", + "thumb_360": "https://files.slack.com/files-tmb/T01CWUYP5AR-F05SUDUQEDN-b8a4896bfd/screenshot_2023-09-13_at_5.31.22_pm_360.png", + "thumb_360_w": 360, + "thumb_360_h": 161, + "thumb_480": "https://files.slack.com/files-tmb/T01CWUYP5AR-F05SUDUQEDN-b8a4896bfd/screenshot_2023-09-13_at_5.31.22_pm_480.png", + "thumb_480_w": 480, + "thumb_480_h": 214, + "thumb_160": "https://files.slack.com/files-tmb/T01CWUYP5AR-F05SUDUQEDN-b8a4896bfd/screenshot_2023-09-13_at_5.31.22_pm_160.png", + "thumb_720": "https://files.slack.com/files-tmb/T01CWUYP5AR-F05SUDUQEDN-b8a4896bfd/screenshot_2023-09-13_at_5.31.22_pm_720.png", + "thumb_720_w": 720, + "thumb_720_h": 321, + "thumb_800": "https://files.slack.com/files-tmb/T01CWUYP5AR-F05SUDUQEDN-b8a4896bfd/screenshot_2023-09-13_at_5.31.22_pm_800.png", + "thumb_800_w": 800, + "thumb_800_h": 357, + "thumb_960": "https://files.slack.com/files-tmb/T01CWUYP5AR-F05SUDUQEDN-b8a4896bfd/screenshot_2023-09-13_at_5.31.22_pm_960.png", + "thumb_960_w": 960, + "thumb_960_h": 428, + "thumb_1024": "https://files.slack.com/files-tmb/T01CWUYP5AR-F05SUDUQEDN-b8a4896bfd/screenshot_2023-09-13_at_5.31.22_pm_1024.png", + "thumb_1024_w": 1024, + "thumb_1024_h": 457, + "original_w": 3010, + "original_h": 1342, + "thumb_tiny": "AwAVADDS74xQRR/FQaAEpaTtS0AFFLRQAY5pGOKWmv2oATOaXNNHWnUxC5ozSUUDP//Z", + "permalink": "https://openlineage.slack.com/files/U05QL7LN2GH/F05SUDUQEDN/screenshot_2023-09-13_at_5.31.22_pm.png", + "permalink_public": "https://slack-files.com/T01CWUYP5AR-F05SUDUQEDN-edf19dc4d6", + "is_starred": false, + "has_rich_preview": false, + "file_access": "visible" + } + ], + "upload": false, + "user": "U05QL7LN2GH", + "display_as_bot": false, + "x_files": [ + "F05SUDUQEDN" + ], + "ts": "1694606609.510489", + "blocks": [ + { + "type": "rich_text", + "block_id": "SPBs9", + "elements": [ + { + "type": "rich_text_section", + "elements": [ + { + "type": "link", + "url": "https://openlineage.slack.com/files/U05QL7LN2GH/F05SUDUQEDN/screenshot_2023-09-13_at_5.31.22_pm.png" + } + ] + } + ] + } + ], + "client_msg_id": "ff1b8b2d-ca2f-4acc-bfdc-a5f76ac0e043", + "thread_ts": "1694545905.974339", + "parent_user_id": "U05QL7LN2GH" + }, + { + "client_msg_id": "eb91ae90-5582-4966-bb4a-b9f875bf6c02", + "type": "message", + "text": "I see. There are few possible things to do here - composer could mount the user files, Airflow could not start plugins on triggerer, or we could detect we're on triggerer and not import anything there. However, does it impact OL or Airflow operation in other way than this log?", + "user": "U01RA9B5GG2", + "ts": "1694607032.263479", + "blocks": [ + { + "type": "rich_text", + "block_id": "YE+8k", + "elements": [ + { + "type": "rich_text_section", + "elements": [ + { + "type": "text", + "text": "I see. There are few possible things to do here - composer could mount the user files, Airflow could not start plugins on triggerer, or we could detect we're on triggerer and not import anything there. However, does it impact OL or Airflow operation in other way than this log?" + } + ] + } + ] + } + ], + "team": "T01CWUYP5AR", + "thread_ts": "1694545905.974339", + "parent_user_id": "U05QL7LN2GH" + }, + { + "client_msg_id": "05bafc1e-0f4b-42bb-919e-24dca030f7af", + "type": "message", + "text": "Probably we'd have to do something if that really bothers you as there won't be further changes to Airflow 2.5", + "user": "U01RA9B5GG2", + "ts": "1694607126.289379", + "blocks": [ + { + "type": "rich_text", + "block_id": "4YD1+", + "elements": [ + { + "type": "rich_text_section", + "elements": [ + { + "type": "text", + "text": "Probably we'd have to do something if that really bothers you as there won't be further changes to Airflow 2.5" + } + ] + } + ] + } + ], + "team": "T01CWUYP5AR", + "thread_ts": "1694545905.974339", + "parent_user_id": "U05QL7LN2GH" + }, + { + "client_msg_id": "48836a79-3a5e-4376-aebe-55d30bc8413b", + "type": "message", + "text": "The Problem is it is actually not registering this custom extractor written by me, henceforth i am just receiving the DefaultExtractor things and my piece of extractor code is not even getting triggered", + "user": "U05QL7LN2GH", + "ts": "1694607494.517249", + "blocks": [ + { + "type": "rich_text", + "block_id": "2ufg6", + "elements": [ + { + "type": "rich_text_section", + "elements": [ + { + "type": "text", + "text": "The Problem is it is actually not registering this custom extractor written by me, henceforth i am just receiving the DefaultExtractor things and my piece of extractor code is not even getting triggered" + } + ] + } + ] + } + ], + "team": "T01CWUYP5AR", + "thread_ts": "1694545905.974339", + "parent_user_id": "U05QL7LN2GH" + }, + { + "client_msg_id": "c3b668ca-b149-453b-ac13-992613f90f61", + "type": "message", + "text": "any suggestions to try <@U01RA9B5GG2>", + "user": "U05QL7LN2GH", + "ts": "1694607769.601109", + "blocks": [ + { + "type": "rich_text", + "block_id": "PM4tm", + "elements": [ + { + "type": "rich_text_section", + "elements": [ + { + "type": "text", + "text": "any suggestions to try " + }, + { + "type": "user", + "user_id": "U01RA9B5GG2" + } + ] + } + ] + } + ], + "team": "T01CWUYP5AR", + "thread_ts": "1694545905.974339", + "parent_user_id": "U05QL7LN2GH" + }, + { + "client_msg_id": "4435ba4e-2446-4d1d-9426-8fd5035e332d", + "type": "message", + "text": "Could you share worker logs?", + "user": "U01RA9B5GG2", + "ts": "1694608068.429439", + "blocks": [ + { + "type": "rich_text", + "block_id": "k/5hQ", + "elements": [ + { + "type": "rich_text_section", + "elements": [ + { + "type": "text", + "text": "Could you share worker logs?" + } + ] + } + ] + } + ], + "team": "T01CWUYP5AR", + "thread_ts": "1694545905.974339", + "parent_user_id": "U05QL7LN2GH" + }, + { + "client_msg_id": "ae8653c1-f153-447d-b791-aa4fe703d5b9", + "type": "message", + "text": "and check if module is importable from your dag code?", + "user": "U01RA9B5GG2", + "ts": "1694608076.879469", + "blocks": [ + { + "type": "rich_text", + "block_id": "B+JX3", + "elements": [ + { + "type": "rich_text_section", + "elements": [ + { + "type": "text", + "text": "and check if module is importable from your dag code?" + } + ] + } + ] + } + ], + "team": "T01CWUYP5AR", + "thread_ts": "1694545905.974339", + "parent_user_id": "U05QL7LN2GH" + }, + { + "type": "message", + "text": "these are the worker pod logs…where there is no log of openlineageplugin", + "files": [ + { + "id": "F05S5J5AY2E", + "created": 1694608245, + "timestamp": 1694608245, + "name": "downloaded-logs-20230913-180017.json", + "title": "downloaded-logs-20230913-180017.json", + "mimetype": "text/plain", + "filetype": "json", + "pretty_type": "JSON", + "user": "U05QL7LN2GH", + "user_team": "T01CWUYP5AR", + "editable": false, + "size": 2886952, + "mode": "hosted", + "is_external": false, + "external_type": "", + "is_public": true, + "public_url_shared": false, + "display_as_bot": false, + "username": "", + "url_private": "https://files.slack.com/files-pri/T01CWUYP5AR-F05S5J5AY2E/downloaded-logs-20230913-180017.json", + "url_private_download": "https://files.slack.com/files-pri/T01CWUYP5AR-F05S5J5AY2E/download/downloaded-logs-20230913-180017.json", + "media_display_type": "unknown", + "permalink": "https://openlineage.slack.com/files/U05QL7LN2GH/F05S5J5AY2E/downloaded-logs-20230913-180017.json", + "permalink_public": "https://slack-files.com/T01CWUYP5AR-F05S5J5AY2E-8c5402e3d8", + "is_starred": false, + "has_rich_preview": false, + "file_access": "visible" + } + ], + "upload": false, + "user": "U05QL7LN2GH", + "display_as_bot": false, + "ts": "1694608285.567279", + "blocks": [ + { + "type": "rich_text", + "block_id": "nA/mA", + "elements": [ + { + "type": "rich_text_section", + "elements": [ + { + "type": "text", + "text": "these are the worker pod logs…where there is no log of openlineageplugin" + } + ] + } + ] + } + ], + "client_msg_id": "31f0c42a-f1a4-4064-851d-09d3da4a1c00", + "thread_ts": "1694545905.974339", + "parent_user_id": "U05QL7LN2GH" + }, + { + "client_msg_id": "4332faa1-bbb5-4073-bd60-7e9865482513", + "type": "message", + "text": " --> sure will check now on this one", + "user": "U05QL7LN2GH", + "ts": "1694608312.873729", + "blocks": [ + { + "type": "rich_text", + "block_id": "jM2YJ", + "elements": [ + { + "type": "rich_text_section", + "elements": [ + { + "type": "link", + "url": "https://openlineage.slack.com/archives/C01CK9T7HKR/p1694608076879469?thread_ts=1694545905.974339&cid=C01CK9T7HKR" + }, + { + "type": "text", + "text": " --> sure will check now on this one" + } + ] + } + ] + } + ], + "team": "T01CWUYP5AR", + "attachments": [ + { + "from_url": "https://openlineage.slack.com/archives/C01CK9T7HKR/p1694608076879469?thread_ts=1694545905.974339&cid=C01CK9T7HKR", + "ts": "1694608076.879469", + "author_id": "U01RA9B5GG2", + "channel_id": "C01CK9T7HKR", + "channel_team": "T01CWUYP5AR", + "is_msg_unfurl": true, + "is_reply_unfurl": true, + "message_blocks": [ + { + "team": "T01CWUYP5AR", + "channel": "C01CK9T7HKR", + "ts": "1694608076.879469", + "message": { + "blocks": [ + { + "type": "rich_text", + "block_id": "B+JX3", + "elements": [ + { + "type": "rich_text_section", + "elements": [ + { + "type": "text", + "text": "and check if module is importable from your dag code?" + } + ] + } + ] + } + ] + } + } + ], + "id": 1, + "original_url": "https://openlineage.slack.com/archives/C01CK9T7HKR/p1694608076879469?thread_ts=1694545905.974339&cid=C01CK9T7HKR", + "fallback": "[September 13th, 2023 5:27 AM] maciej.obuchowski: and check if module is importable from your dag code?", + "text": "and check if module is importable from your dag code?", + "author_name": "Maciej Obuchowski", + "author_link": "https://openlineage.slack.com/team/U01RA9B5GG2", + "author_icon": "https://avatars.slack-edge.com/2021-03-16/1888464110336_2f556751039c2a20f1fe_48.jpg", + "author_subname": "Maciej Obuchowski", + "mrkdwn_in": [ + "text" + ], + "footer": "Thread in Slack Conversation" + } + ], + "thread_ts": "1694545905.974339", + "parent_user_id": "U05QL7LN2GH" + }, + { + "client_msg_id": "ba6d946f-3279-44dc-963e-63f8d426691e", + "type": "message", + "text": "``` {\n \"textPayload\": \"Traceback (most recent call last): File \\\"/opt/python3.8/lib/python3.8/site-packages/openlineage/airflow/utils.py\\\", line 427, in import_from_string module = importlib.import_module(module_path) File \\\"/opt/python3.8/lib/python3.8/importlib/__init__.py\\\", line 127, in import_module return _bootstrap._gcd_import(name[level:], package, level) File \\\"<frozen importlib._bootstrap>\\\", line 1014, in _gcd_import File \\\"<frozen importlib._bootstrap>\\\", line 991, in _find_and_load File \\\"<frozen importlib._bootstrap>\\\", line 961, in _find_and_load_unlocked File \\\"<frozen importlib._bootstrap>\\\", line 219, in _call_with_frames_removed File \\\"<frozen importlib._bootstrap>\\\", line 1014, in _gcd_import File \\\"<frozen importlib._bootstrap>\\\", line 991, in _find_and_load File \\\"<frozen importlib._bootstrap>\\\", line 961, in _find_and_load_unlocked File \\\"<frozen importlib._bootstrap>\\\", line 219, in _call_with_frames_removed File \\\"<frozen importlib._bootstrap>\\\", line 1014, in _gcd_import File \\\"<frozen importlib._bootstrap>\\\", line 991, in _find_and_load File \\\"<frozen importlib._bootstrap>\\\", line 973, in _find_and_load_unlockedModuleNotFoundError: No module named 'airflow.gcs'\",\n \"insertId\": \"pt2eu6fl9z5vw\",\n \"resource\": {\n \"type\": \"cloud_composer_environment\",\n \"labels\": {\n \"environment_name\": \"openlineage\",\n \"location\": \"us-west1\",\n \"project_id\": \"acceldata-acm\"\n }\n },\n \"timestamp\": \"2023-09-13T06:20:44.131577764Z\",\n \"severity\": \"ERROR\",\n \"labels\": {\n \"worker_id\": \"airflow-worker-xttt8\"\n },\n \"logName\": \"projects/acceldata-acm/logs/airflow-worker\",\n \"receiveTimestamp\": \"2023-09-13T06:20:48.847319607Z\"\n },```\nit doesn't see `No module named 'airflow.gcs'` that is part of your extractor path `airflow.gcs.dags.big_query_insert_job_extractor.BigQueryInsertJobExtractor`\nhowever, is it necessary? I generally see people using imports directly from dags folder", + "user": "U01RA9B5GG2", + "ts": "1694608712.776939", + "blocks": [ + { + "type": "rich_text", + "block_id": "SNTlV", + "elements": [ + { + "type": "rich_text_preformatted", + "elements": [ + { + "type": "text", + "text": " {\n \"textPayload\": \"Traceback (most recent call last): File \\\"/opt/python3.8/lib/python3.8/site-packages/openlineage/airflow/utils.py\\\", line 427, in import_from_string module = importlib.import_module(module_path) File \\\"/opt/python3.8/lib/python3.8/importlib/__init__.py\\\", line 127, in import_module return _bootstrap._gcd_import(name[level:], package, level) File \\\"\\\", line 1014, in _gcd_import File \\\"\\\", line 991, in _find_and_load File \\\"\\\", line 961, in _find_and_load_unlocked File \\\"\\\", line 219, in _call_with_frames_removed File \\\"\\\", line 1014, in _gcd_import File \\\"\\\", line 991, in _find_and_load File \\\"\\\", line 961, in _find_and_load_unlocked File \\\"\\\", line 219, in _call_with_frames_removed File \\\"\\\", line 1014, in _gcd_import File \\\"\\\", line 991, in _find_and_load File \\\"\\\", line 973, in _find_and_load_unlockedModuleNotFoundError: No module named 'airflow.gcs'\",\n \"insertId\": \"pt2eu6fl9z5vw\",\n \"resource\": {\n \"type\": \"cloud_composer_environment\",\n \"labels\": {\n \"environment_name\": \"openlineage\",\n \"location\": \"us-west1\",\n \"project_id\": \"acceldata-acm\"\n }\n },\n \"timestamp\": \"2023-09-13T06:20:44.131577764Z\",\n \"severity\": \"ERROR\",\n \"labels\": {\n \"worker_id\": \"airflow-worker-xttt8\"\n },\n \"logName\": \"projects/acceldata-acm/logs/airflow-worker\",\n \"receiveTimestamp\": \"2023-09-13T06:20:48.847319607Z\"\n }," + } + ], + "border": 0 + }, + { + "type": "rich_text_section", + "elements": [ + { + "type": "text", + "text": "\nit doesn't see " + }, + { + "type": "text", + "text": "No module named 'airflow.gcs'", + "style": { + "code": true + } + }, + { + "type": "text", + "text": " that is part of your extractor path " + }, + { + "type": "text", + "text": "airflow.gcs.dags.big_query_insert_job_extractor.BigQueryInsertJobExtractor", + "style": { + "code": true + } + }, + { + "type": "text", + "text": "\nhowever, is it necessary? I generally see people using imports directly from dags folder" + } + ] + } + ] + } + ], + "team": "T01CWUYP5AR", + "edited": { + "user": "U01RA9B5GG2", + "ts": "1694608729.000000" + }, + "thread_ts": "1694545905.974339", + "parent_user_id": "U05QL7LN2GH" + }, + { + "type": "message", + "text": "this is one of the experimentation that i have did, but then i reverted it back to keeping it to dependencies.big_query_insert_job_extractor.BigQueryInsertJobExtractor…where dependencies is a module i have created inside my dags folder", + "files": [ + { + "id": "F05S2NCLM9T", + "created": 1694609020, + "timestamp": 1694609020, + "name": "Screenshot 2023-09-13 at 6.13.36 PM.png", + "title": "Screenshot 2023-09-13 at 6.13.36 PM.png", + "mimetype": "image/png", + "filetype": "png", + "pretty_type": "PNG", + "user": "U05QL7LN2GH", + "user_team": "T01CWUYP5AR", + "editable": false, + "size": 55433, + "mode": "hosted", + "is_external": false, + "external_type": "", + "is_public": true, + "public_url_shared": false, + "display_as_bot": false, + "username": "", + "url_private": "https://files.slack.com/files-pri/T01CWUYP5AR-F05S2NCLM9T/screenshot_2023-09-13_at_6.13.36_pm.png", + "url_private_download": "https://files.slack.com/files-pri/T01CWUYP5AR-F05S2NCLM9T/download/screenshot_2023-09-13_at_6.13.36_pm.png", + "media_display_type": "unknown", + "thumb_64": "https://files.slack.com/files-tmb/T01CWUYP5AR-F05S2NCLM9T-09572876f7/screenshot_2023-09-13_at_6.13.36_pm_64.png", + "thumb_80": "https://files.slack.com/files-tmb/T01CWUYP5AR-F05S2NCLM9T-09572876f7/screenshot_2023-09-13_at_6.13.36_pm_80.png", + "thumb_360": "https://files.slack.com/files-tmb/T01CWUYP5AR-F05S2NCLM9T-09572876f7/screenshot_2023-09-13_at_6.13.36_pm_360.png", + "thumb_360_w": 360, + "thumb_360_h": 79, + "thumb_480": "https://files.slack.com/files-tmb/T01CWUYP5AR-F05S2NCLM9T-09572876f7/screenshot_2023-09-13_at_6.13.36_pm_480.png", + "thumb_480_w": 480, + "thumb_480_h": 105, + "thumb_160": "https://files.slack.com/files-tmb/T01CWUYP5AR-F05S2NCLM9T-09572876f7/screenshot_2023-09-13_at_6.13.36_pm_160.png", + "thumb_720": "https://files.slack.com/files-tmb/T01CWUYP5AR-F05S2NCLM9T-09572876f7/screenshot_2023-09-13_at_6.13.36_pm_720.png", + "thumb_720_w": 720, + "thumb_720_h": 158, + "thumb_800": "https://files.slack.com/files-tmb/T01CWUYP5AR-F05S2NCLM9T-09572876f7/screenshot_2023-09-13_at_6.13.36_pm_800.png", + "thumb_800_w": 800, + "thumb_800_h": 175, + "thumb_960": "https://files.slack.com/files-tmb/T01CWUYP5AR-F05S2NCLM9T-09572876f7/screenshot_2023-09-13_at_6.13.36_pm_960.png", + "thumb_960_w": 960, + "thumb_960_h": 210, + "thumb_1024": "https://files.slack.com/files-tmb/T01CWUYP5AR-F05S2NCLM9T-09572876f7/screenshot_2023-09-13_at_6.13.36_pm_1024.png", + "thumb_1024_w": 1024, + "thumb_1024_h": 224, + "original_w": 1270, + "original_h": 278, + "thumb_tiny": "AwAKADDRyc460vNLRQA0McUuT6UtFACc+1HPtS0UAf/Z", + "permalink": "https://openlineage.slack.com/files/U05QL7LN2GH/F05S2NCLM9T/screenshot_2023-09-13_at_6.13.36_pm.png", + "permalink_public": "https://slack-files.com/T01CWUYP5AR-F05S2NCLM9T-af73e2f6b1", + "is_starred": false, + "has_rich_preview": false, + "file_access": "visible" + } + ], + "upload": false, + "user": "U05QL7LN2GH", + "display_as_bot": false, + "ts": "1694609051.132379", + "blocks": [ + { + "type": "rich_text", + "block_id": "lfYf2", + "elements": [ + { + "type": "rich_text_section", + "elements": [ + { + "type": "text", + "text": "this is one of the experimentation that i have did, but then i reverted it back to keeping it to dependencies.big_query_insert_job_extractor.BigQueryInsertJobExtractor…where dependencies is a module i have created inside my dags folder" + } + ] + } + ] + } + ], + "client_msg_id": "5815ad59-e69c-4c15-8610-a8117ceb31c2", + "thread_ts": "1694545905.974339", + "parent_user_id": "U05QL7LN2GH" + }, + { + "type": "message", + "text": "", + "files": [ + { + "id": "F05RM6EV6DV", + "created": 1694545739, + "timestamp": 1694545739, + "name": "Screenshot 2023-09-13 at 12.38.55 AM.png", + "title": "Screenshot 2023-09-13 at 12.38.55 AM.png", + "mimetype": "image/png", + "filetype": "png", + "pretty_type": "PNG", + "user": "U05QL7LN2GH", + "user_team": "T01CWUYP5AR", + "editable": false, + "size": 951188, + "mode": "hosted", + "is_external": false, + "external_type": "", + "is_public": true, + "public_url_shared": false, + "display_as_bot": false, + "username": "", + "url_private": "https://files.slack.com/files-pri/T01CWUYP5AR-F05RM6EV6DV/screenshot_2023-09-13_at_12.38.55_am.png", + "url_private_download": "https://files.slack.com/files-pri/T01CWUYP5AR-F05RM6EV6DV/download/screenshot_2023-09-13_at_12.38.55_am.png", + "media_display_type": "unknown", + "thumb_64": "https://files.slack.com/files-tmb/T01CWUYP5AR-F05RM6EV6DV-e20cfb50c7/screenshot_2023-09-13_at_12.38.55_am_64.png", + "thumb_80": "https://files.slack.com/files-tmb/T01CWUYP5AR-F05RM6EV6DV-e20cfb50c7/screenshot_2023-09-13_at_12.38.55_am_80.png", + "thumb_360": "https://files.slack.com/files-tmb/T01CWUYP5AR-F05RM6EV6DV-e20cfb50c7/screenshot_2023-09-13_at_12.38.55_am_360.png", + "thumb_360_w": 360, + "thumb_360_h": 122, + "thumb_480": "https://files.slack.com/files-tmb/T01CWUYP5AR-F05RM6EV6DV-e20cfb50c7/screenshot_2023-09-13_at_12.38.55_am_480.png", + "thumb_480_w": 480, + "thumb_480_h": 162, + "thumb_160": "https://files.slack.com/files-tmb/T01CWUYP5AR-F05RM6EV6DV-e20cfb50c7/screenshot_2023-09-13_at_12.38.55_am_160.png", + "thumb_720": "https://files.slack.com/files-tmb/T01CWUYP5AR-F05RM6EV6DV-e20cfb50c7/screenshot_2023-09-13_at_12.38.55_am_720.png", + "thumb_720_w": 720, + "thumb_720_h": 243, + "thumb_800": "https://files.slack.com/files-tmb/T01CWUYP5AR-F05RM6EV6DV-e20cfb50c7/screenshot_2023-09-13_at_12.38.55_am_800.png", + "thumb_800_w": 800, + "thumb_800_h": 270, + "thumb_960": "https://files.slack.com/files-tmb/T01CWUYP5AR-F05RM6EV6DV-e20cfb50c7/screenshot_2023-09-13_at_12.38.55_am_960.png", + "thumb_960_w": 960, + "thumb_960_h": 324, + "thumb_1024": "https://files.slack.com/files-tmb/T01CWUYP5AR-F05RM6EV6DV-e20cfb50c7/screenshot_2023-09-13_at_12.38.55_am_1024.png", + "thumb_1024_w": 1024, + "thumb_1024_h": 346, + "original_w": 3024, + "original_h": 1022, + "thumb_tiny": "AwAQADDRx2pefWg5zRQAtFJ+FHNAC0YpMmjmgD//2Q==", + "permalink": "https://openlineage.slack.com/files/U05QL7LN2GH/F05RM6EV6DV/screenshot_2023-09-13_at_12.38.55_am.png", + "permalink_public": "https://slack-files.com/T01CWUYP5AR-F05RM6EV6DV-62656c8fb4", + "is_starred": false, + "has_rich_preview": false, + "file_access": "visible" + } + ], + "upload": false, + "user": "U05QL7LN2GH", + "display_as_bot": false, + "x_files": [ + "F05RM6EV6DV" + ], + "ts": "1694609073.130239", + "blocks": [ + { + "type": "rich_text", + "block_id": "BGYlB", + "elements": [ + { + "type": "rich_text_section", + "elements": [ + { + "type": "link", + "url": "https://openlineage.slack.com/files/U05QL7LN2GH/F05RM6EV6DV/screenshot_2023-09-13_at_12.38.55_am.png" + } + ] + } + ] + } + ], + "client_msg_id": "457a61b4-706d-433a-b375-bf640843c913", + "thread_ts": "1694545905.974339", + "parent_user_id": "U05QL7LN2GH" + }, + { + "type": "message", + "text": "these are the logs of the triggerer pod specifically", + "files": [ + { + "id": "F05S2NNQ1UM", + "created": 1694609135, + "timestamp": 1694609135, + "name": "Screenshot 2023-09-13 at 6.15.30 PM.png", + "title": "Screenshot 2023-09-13 at 6.15.30 PM.png", + "mimetype": "image/png", + "filetype": "png", + "pretty_type": "PNG", + "user": "U05QL7LN2GH", + "user_team": "T01CWUYP5AR", + "editable": false, + "size": 1349379, + "mode": "hosted", + "is_external": false, + "external_type": "", + "is_public": true, + "public_url_shared": false, + "display_as_bot": false, + "username": "", + "url_private": "https://files.slack.com/files-pri/T01CWUYP5AR-F05S2NNQ1UM/screenshot_2023-09-13_at_6.15.30_pm.png", + "url_private_download": "https://files.slack.com/files-pri/T01CWUYP5AR-F05S2NNQ1UM/download/screenshot_2023-09-13_at_6.15.30_pm.png", + "media_display_type": "unknown", + "thumb_64": "https://files.slack.com/files-tmb/T01CWUYP5AR-F05S2NNQ1UM-7870a10644/screenshot_2023-09-13_at_6.15.30_pm_64.png", + "thumb_80": "https://files.slack.com/files-tmb/T01CWUYP5AR-F05S2NNQ1UM-7870a10644/screenshot_2023-09-13_at_6.15.30_pm_80.png", + "thumb_360": "https://files.slack.com/files-tmb/T01CWUYP5AR-F05S2NNQ1UM-7870a10644/screenshot_2023-09-13_at_6.15.30_pm_360.png", + "thumb_360_w": 360, + "thumb_360_h": 162, + "thumb_480": "https://files.slack.com/files-tmb/T01CWUYP5AR-F05S2NNQ1UM-7870a10644/screenshot_2023-09-13_at_6.15.30_pm_480.png", + "thumb_480_w": 480, + "thumb_480_h": 217, + "thumb_160": "https://files.slack.com/files-tmb/T01CWUYP5AR-F05S2NNQ1UM-7870a10644/screenshot_2023-09-13_at_6.15.30_pm_160.png", + "thumb_720": "https://files.slack.com/files-tmb/T01CWUYP5AR-F05S2NNQ1UM-7870a10644/screenshot_2023-09-13_at_6.15.30_pm_720.png", + "thumb_720_w": 720, + "thumb_720_h": 325, + "thumb_800": "https://files.slack.com/files-tmb/T01CWUYP5AR-F05S2NNQ1UM-7870a10644/screenshot_2023-09-13_at_6.15.30_pm_800.png", + "thumb_800_w": 800, + "thumb_800_h": 361, + "thumb_960": "https://files.slack.com/files-tmb/T01CWUYP5AR-F05S2NNQ1UM-7870a10644/screenshot_2023-09-13_at_6.15.30_pm_960.png", + "thumb_960_w": 960, + "thumb_960_h": 433, + "thumb_1024": "https://files.slack.com/files-tmb/T01CWUYP5AR-F05S2NNQ1UM-7870a10644/screenshot_2023-09-13_at_6.15.30_pm_1024.png", + "thumb_1024_w": 1024, + "thumb_1024_h": 462, + "original_w": 3018, + "original_h": 1362, + "thumb_tiny": "AwAVADCpz60hHvS0Z9qAG496SlNJigAooooAcDRmkpaAENJSmkPWgBM0ZoooA//Z", + "permalink": "https://openlineage.slack.com/files/U05QL7LN2GH/F05S2NNQ1UM/screenshot_2023-09-13_at_6.15.30_pm.png", + "permalink_public": "https://slack-files.com/T01CWUYP5AR-F05S2NNQ1UM-9a548d0a76", + "is_starred": false, + "has_rich_preview": false, + "file_access": "visible" + } + ], + "upload": false, + "user": "U05QL7LN2GH", + "display_as_bot": false, + "ts": "1694609146.705369", + "blocks": [ + { + "type": "rich_text", + "block_id": "SHNkd", + "elements": [ + { + "type": "rich_text_section", + "elements": [ + { + "type": "text", + "text": "these are the logs of the triggerer pod specifically" + } + ] + } + ] + } + ], + "client_msg_id": "47c532c4-9680-4dff-9073-84fd361f08cc", + "thread_ts": "1694545905.974339", + "parent_user_id": "U05QL7LN2GH" + }, + { + "client_msg_id": "3032c52c-7420-4f63-9c06-9b33633e03a3", + "type": "message", + "text": "yeah it would be expected to have this in triggerer where it's not mounted, but will it behave the same for worker where it's mounted?", + "user": "U01RA9B5GG2", + "ts": "1694609191.704139", + "blocks": [ + { + "type": "rich_text", + "block_id": "iH3ag", + "elements": [ + { + "type": "rich_text_section", + "elements": [ + { + "type": "text", + "text": "yeah it would be expected to have this in triggerer where it's not mounted, but will it behave the same for worker where it's mounted?" + } + ] + } + ] + } + ], + "team": "T01CWUYP5AR", + "thread_ts": "1694545905.974339", + "parent_user_id": "U05QL7LN2GH" + }, + { + "client_msg_id": "db50bbab-6b72-43a1-b68f-db0eb7052cf0", + "type": "message", + "text": "maybe `___init___.py` is missing for top-level dag path?", + "user": "U01RA9B5GG2", + "ts": "1694609229.577469", + "blocks": [ + { + "type": "rich_text", + "block_id": "7+IIh", + "elements": [ + { + "type": "rich_text_section", + "elements": [ + { + "type": "text", + "text": "maybe " + }, + { + "type": "text", + "text": "__", + "style": { + "code": true + } + }, + { + "type": "text", + "text": "init", + "style": { + "italic": true, + "code": true + } + }, + { + "type": "text", + "text": "__.py", + "style": { + "code": true + } + }, + { + "type": "text", + "text": " is missing for top-level dag path?" + } + ] + } + ] + } + ], + "team": "T01CWUYP5AR", + "thread_ts": "1694545905.974339", + "parent_user_id": "U05QL7LN2GH" + }, + { + "type": "message", + "text": "these are the logs of the worker pod at startup, where it does not complain of the plugin like in triggerer, but when tasks are run on this worker…somehow it is not picking up the extractor for the operator that i have written it for", + "files": [ + { + "id": "F05RR4YHH1V", + "created": 1694609270, + "timestamp": 1694609270, + "name": "Screenshot 2023-09-13 at 6.17.46 PM.png", + "title": "Screenshot 2023-09-13 at 6.17.46 PM.png", + "mimetype": "image/png", + "filetype": "png", + "pretty_type": "PNG", + "user": "U05QL7LN2GH", + "user_team": "T01CWUYP5AR", + "editable": false, + "size": 1106398, + "mode": "hosted", + "is_external": false, + "external_type": "", + "is_public": true, + "public_url_shared": false, + "display_as_bot": false, + "username": "", + "url_private": "https://files.slack.com/files-pri/T01CWUYP5AR-F05RR4YHH1V/screenshot_2023-09-13_at_6.17.46_pm.png", + "url_private_download": "https://files.slack.com/files-pri/T01CWUYP5AR-F05RR4YHH1V/download/screenshot_2023-09-13_at_6.17.46_pm.png", + "media_display_type": "unknown", + "thumb_64": "https://files.slack.com/files-tmb/T01CWUYP5AR-F05RR4YHH1V-63b8fc9fb8/screenshot_2023-09-13_at_6.17.46_pm_64.png", + "thumb_80": "https://files.slack.com/files-tmb/T01CWUYP5AR-F05RR4YHH1V-63b8fc9fb8/screenshot_2023-09-13_at_6.17.46_pm_80.png", + "thumb_360": "https://files.slack.com/files-tmb/T01CWUYP5AR-F05RR4YHH1V-63b8fc9fb8/screenshot_2023-09-13_at_6.17.46_pm_360.png", + "thumb_360_w": 360, + "thumb_360_h": 175, + "thumb_480": "https://files.slack.com/files-tmb/T01CWUYP5AR-F05RR4YHH1V-63b8fc9fb8/screenshot_2023-09-13_at_6.17.46_pm_480.png", + "thumb_480_w": 480, + "thumb_480_h": 234, + "thumb_160": "https://files.slack.com/files-tmb/T01CWUYP5AR-F05RR4YHH1V-63b8fc9fb8/screenshot_2023-09-13_at_6.17.46_pm_160.png", + "thumb_720": "https://files.slack.com/files-tmb/T01CWUYP5AR-F05RR4YHH1V-63b8fc9fb8/screenshot_2023-09-13_at_6.17.46_pm_720.png", + "thumb_720_w": 720, + "thumb_720_h": 350, + "thumb_800": "https://files.slack.com/files-tmb/T01CWUYP5AR-F05RR4YHH1V-63b8fc9fb8/screenshot_2023-09-13_at_6.17.46_pm_800.png", + "thumb_800_w": 800, + "thumb_800_h": 389, + "thumb_960": "https://files.slack.com/files-tmb/T01CWUYP5AR-F05RR4YHH1V-63b8fc9fb8/screenshot_2023-09-13_at_6.17.46_pm_960.png", + "thumb_960_w": 960, + "thumb_960_h": 467, + "thumb_1024": "https://files.slack.com/files-tmb/T01CWUYP5AR-F05RR4YHH1V-63b8fc9fb8/screenshot_2023-09-13_at_6.17.46_pm_1024.png", + "thumb_1024_w": 1024, + "thumb_1024_h": 498, + "original_w": 3020, + "original_h": 1470, + "thumb_tiny": "AwAXADCpTT160+oz1oAM0UUUAFFFFAC0vFIOho7UABx70lB60UAFFJRQB//Z", + "permalink": "https://openlineage.slack.com/files/U05QL7LN2GH/F05RR4YHH1V/screenshot_2023-09-13_at_6.17.46_pm.png", + "permalink_public": "https://slack-files.com/T01CWUYP5AR-F05RR4YHH1V-7d6ea17c5a", + "is_starred": false, + "has_rich_preview": false, + "file_access": "visible" + } + ], + "upload": false, + "user": "U05QL7LN2GH", + "display_as_bot": false, + "ts": "1694609341.476869", + "blocks": [ + { + "type": "rich_text", + "block_id": "BVyLS", + "elements": [ + { + "type": "rich_text_section", + "elements": [ + { + "type": "text", + "text": "these are the logs of the worker pod at startup, where it does not complain of the plugin like in triggerer, but when tasks are run on this worker…somehow it is not picking up the extractor for the operator that i have written it for" + } + ] + } + ] + } + ], + "client_msg_id": "b2c18b33-ea4b-4010-b787-3e371fc708c5", + "thread_ts": "1694545905.974339", + "parent_user_id": "U05QL7LN2GH" + }, + { + "client_msg_id": "46021901-2df0-4d3e-9c30-a09253aa8dbd", + "type": "message", + "text": " --> you mean to make the dags folder as well like a module by adding the init.py?", + "user": "U05QL7LN2GH", + "ts": "1694609394.103649", + "blocks": [ + { + "type": "rich_text", + "block_id": "ErKg8", + "elements": [ + { + "type": "rich_text_section", + "elements": [ + { + "type": "link", + "url": "https://openlineage.slack.com/archives/C01CK9T7HKR/p1694609229577469?thread_ts=1694545905.974339&cid=C01CK9T7HKR" + }, + { + "type": "text", + "text": " --> you mean to make the dags folder as well like a module by adding the init.py?" + } + ] + } + ] + } + ], + "team": "T01CWUYP5AR", + "edited": { + "user": "U05QL7LN2GH", + "ts": "1694609406.000000" + }, + "attachments": [ + { + "from_url": "https://openlineage.slack.com/archives/C01CK9T7HKR/p1694609229577469?thread_ts=1694545905.974339&cid=C01CK9T7HKR", + "ts": "1694609229.577469", + "author_id": "U01RA9B5GG2", + "channel_id": "C01CK9T7HKR", + "channel_team": "T01CWUYP5AR", + "is_msg_unfurl": true, + "is_reply_unfurl": true, + "message_blocks": [ + { + "team": "T01CWUYP5AR", + "channel": "C01CK9T7HKR", + "ts": "1694609229.577469", + "message": { + "blocks": [ + { + "type": "rich_text", + "block_id": "7+IIh", + "elements": [ + { + "type": "rich_text_section", + "elements": [ + { + "type": "text", + "text": "maybe " + }, + { + "type": "text", + "text": "__", + "style": { + "code": true + } + }, + { + "type": "text", + "text": "init", + "style": { + "italic": true, + "code": true + } + }, + { + "type": "text", + "text": "__.py", + "style": { + "code": true + } + }, + { + "type": "text", + "text": " is missing for top-level dag path?" + } + ] + } + ] + } + ] + } + } + ], + "id": 1, + "original_url": "https://openlineage.slack.com/archives/C01CK9T7HKR/p1694609229577469?thread_ts=1694545905.974339&cid=C01CK9T7HKR", + "fallback": "[September 13th, 2023 5:47 AM] maciej.obuchowski: maybe `___init___.py` is missing for top-level dag path?", + "text": "maybe `___init___.py` is missing for top-level dag path?", + "author_name": "Maciej Obuchowski", + "author_link": "https://openlineage.slack.com/team/U01RA9B5GG2", + "author_icon": "https://avatars.slack-edge.com/2021-03-16/1888464110336_2f556751039c2a20f1fe_48.jpg", + "author_subname": "Maciej Obuchowski", + "mrkdwn_in": [ + "text" + ], + "footer": "Thread in Slack Conversation" + } + ], + "thread_ts": "1694545905.974339", + "parent_user_id": "U05QL7LN2GH" + }, + { + "client_msg_id": "d35288ee-7c82-48a5-b263-7d727e78e062", + "type": "message", + "text": "yes, I would put whole custom code directly in dags folder, to make sure import paths are the problem", + "user": "U01RA9B5GG2", + "ts": "1694609724.897809", + "blocks": [ + { + "type": "rich_text", + "block_id": "fbJhf", + "elements": [ + { + "type": "rich_text_section", + "elements": [ + { + "type": "text", + "text": "yes, I would put whole custom code directly in dags folder, to make sure import paths are the problem" + } + ] + } + ] + } + ], + "team": "T01CWUYP5AR", + "thread_ts": "1694545905.974339", + "parent_user_id": "U05QL7LN2GH" + }, + { + "client_msg_id": "14066c89-79fc-48d5-816c-c915928e4c27", + "type": "message", + "text": "and would be nice if you could set\n```AIRFLOW__LOGGING__LOGGING_LEVEL=\"DEBUG\"```", + "user": "U01RA9B5GG2", + "ts": "1694609748.555169", + "blocks": [ + { + "type": "rich_text", + "block_id": "FE5kO", + "elements": [ + { + "type": "rich_text_section", + "elements": [ + { + "type": "text", + "text": "and would be nice if you could set\n" + } + ] + }, + { + "type": "rich_text_preformatted", + "elements": [ + { + "type": "text", + "text": "AIRFLOW__LOGGING__LOGGING_LEVEL=\"DEBUG\"" + } + ], + "border": 0 + } + ] + } + ], + "team": "T01CWUYP5AR", + "thread_ts": "1694545905.974339", + "parent_user_id": "U05QL7LN2GH" + }, + { + "client_msg_id": "0b0d7e3c-1c26-44cf-ba66-c1527f6c7308", + "type": "message", + "text": "```Starting the process, got command: triggerer\nInitializing airflow.cfg.\nairflow.cfg initialization is done.\n[2023-09-13T13:11:46.620+0000] {settings.py:267} DEBUG - Setting up DB connection pool (PID 8)\n[2023-09-13T13:11:46.622+0000] {settings.py:372} DEBUG - settings.prepare_engine_args(): Using pool settings. pool_size=5, max_overflow=10, pool_recycle=570, pid=8\n[2023-09-13T13:11:46.742+0000] {cli_action_loggers.py:39} DEBUG - Adding <function default_action_log at 0x7ff39ca1d3a0> to pre execution callback\n[2023-09-13T13:11:47.638+0000] {cli_action_loggers.py:65} DEBUG - Calling callbacks: [<function default_action_log at 0x7ff39ca1d3a0>]\n ____________ _____________\n ____ |__( )_________ __/__ /________ __\n____ /| |_ /__ ___/_ /_ __ /_ __ \\_ | /| / /\n___ ___ | / _ / _ __/ _ / / /_/ /_ |/ |/ /\n _/_/ |_/_/ /_/ /_/ /_/ \\____/____/|__/\n[2023-09-13T13:11:50.527+0000] {plugins_manager.py:300} DEBUG - Loading plugins\n[2023-09-13T13:11:50.580+0000] {plugins_manager.py:244} DEBUG - Loading plugins from directory: /home/airflow/gcs/plugins\n[2023-09-13T13:11:50.581+0000] {plugins_manager.py:224} DEBUG - Loading plugins from entrypoints\n[2023-09-13T13:11:50.587+0000] {plugins_manager.py:227} DEBUG - Importing entry_point plugin OpenLineagePlugin\n[2023-09-13T13:11:50.740+0000] {utils.py:430} WARNING - No module named 'boto3'\n[2023-09-13T13:11:50.743+0000] {utils.py:430} WARNING - No module named 'botocore'\n[2023-09-13T13:11:50.833+0000] {utils.py:430} WARNING - No module named 'airflow.providers.sftp'\n[2023-09-13T13:11:51.144+0000] {utils.py:430} WARNING - No module named 'big_query_insert_job_extractor'\n[2023-09-13T13:11:51.145+0000] {plugins_manager.py:237} ERROR - Failed to import plugin OpenLineagePlugin\nTraceback (most recent call last):\n File \"/opt/python3.8/lib/python3.8/site-packages/openlineage/airflow/utils.py\", line 427, in import_from_string\n module = importlib.import_module(module_path)\n File \"/opt/python3.8/lib/python3.8/importlib/__init__.py\", line 127, in import_module\n return _bootstrap._gcd_import(name[level:], package, level)\n File \"<frozen importlib._bootstrap>\", line 1014, in _gcd_import\n File \"<frozen importlib._bootstrap>\", line 991, in _find_and_load\n File \"<frozen importlib._bootstrap>\", line 973, in _find_and_load_unlocked\nModuleNotFoundError: No module named 'big_query_insert_job_extractor'\n\nThe above exception was the direct cause of the following exception:\n\nTraceback (most recent call last):\n File \"/opt/python3.8/lib/python3.8/site-packages/airflow/plugins_manager.py\", line 229, in load_entrypoint_plugins\n plugin_class = entry_point.load()\n File \"/opt/python3.8/lib/python3.8/site-packages/setuptools/_vendor/importlib_metadata/__init__.py\", line 194, in load\n module = import_module(match.group('module'))\n File \"/opt/python3.8/lib/python3.8/importlib/__init__.py\", line 127, in import_module\n return _bootstrap._gcd_import(name[level:], package, level)\n File \"<frozen importlib._bootstrap>\", line 1014, in _gcd_import\n File \"<frozen importlib._bootstrap>\", line 991, in _find_and_load\n File \"<frozen importlib._bootstrap>\", line 975, in _find_and_load_unlocked\n File \"<frozen importlib._bootstrap>\", line 671, in _load_unlocked\n File \"<frozen importlib._bootstrap_external>\", line 843, in exec_module\n File \"<frozen importlib._bootstrap>\", line 219, in _call_with_frames_removed\n File \"/opt/python3.8/lib/python3.8/site-packages/openlineage/airflow/plugin.py\", line 32, in <module>\n from openlineage.airflow import listener\n File \"/opt/python3.8/lib/python3.8/site-packages/openlineage/airflow/listener.py\", line 75, in <module>\n extractor_manager = ExtractorManager()\n File \"/opt/python3.8/lib/python3.8/site-packages/openlineage/airflow/extractors/manager.py\", line 16, in __init__\n self.task_to_extractor = Extractors()\n File \"/opt/python3.8/lib/python3.8/site-packages/openlineage/airflow/extractors/extractors.py\", line 122, in __init__\n extractor = import_from_string(extractor.strip())\n File \"/opt/python3.8/lib/python3.8/site-packages/openlineage/airflow/utils.py\", line 431, in import_from_string\n raise ImportError(f\"Failed to import {path}\") from e\nImportError: Failed to import big_query_insert_job_extractor.BigQueryInsertJobExtractor\n[2023-09-13T13:11:51.235+0000] {plugins_manager.py:227} DEBUG - Importing entry_point plugin composer_menu_plugin\n[2023-09-13T13:11:51.719+0000] {plugins_manager.py:316} DEBUG - Loading 1 plugin(s) took 1.14 seconds\n[2023-09-13T13:11:51.733+0000] {triggerer_job.py:101} INFO - Starting the triggerer\n[2023-09-13T13:11:51.734+0000] {selector_events.py:59} DEBUG - Using selector: EpollSelector\n[2023-09-13T13:11:56.118+0000] {base_job.py:240} DEBUG - [heartbeat]\n[2023-09-13T13:12:01.359+0000] {base_job.py:240} DEBUG - [heartbeat]\n[2023-09-13T13:12:06.665+0000] {base_job.py:240} DEBUG - [heartbeat]\n[2023-09-13T13:12:11.880+0000] {base_job.py:240} DEBUG - [heartbeat]\n[2023-09-13T13:12:17.098+0000] {base_job.py:240} DEBUG - [heartbeat]\n[2023-09-13T13:12:22.323+0000] {base_job.py:240} DEBUG - [heartbeat]\n[2023-09-13T13:12:27.597+0000] {base_job.py:240} DEBUG - [heartbeat]\n[2023-09-13T13:12:32.826+0000] {base_job.py:240} DEBUG - [heartbeat]\n[2023-09-13T13:12:38.049+0000] {base_job.py:240} DEBUG - [heartbeat]\n[2023-09-13T13:12:43.275+0000] {base_job.py:240} DEBUG - [heartbeat]\n[2023-09-13T13:12:48.509+0000] {base_job.py:240} DEBUG - [heartbeat]\n[2023-09-13T13:12:53.867+0000] {base_job.py:240} DEBUG - [heartbeat]\n[2023-09-13T13:12:59.087+0000] {base_job.py:240} DEBUG - [heartbeat]\n[2023-09-13T13:13:04.300+0000] {base_job.py:240} DEBUG - [heartbeat]\n[2023-09-13T13:13:09.539+0000] {base_job.py:240} DEBUG - [heartbeat]\n[2023-09-13T13:13:14.785+0000] {base_job.py:240} DEBUG - [heartbeat]\n[2023-09-13T13:13:20.007+0000] {base_job.py:240} DEBUG - [heartbeat]\n[2023-09-13T13:13:25.274+0000] {base_job.py:240} DEBUG - [heartbeat]\n[2023-09-13T13:13:30.510+0000] {base_job.py:240} DEBUG - [heartbeat]\n[2023-09-13T13:13:35.729+0000] {base_job.py:240} DEBUG - [heartbeat]\n[2023-09-13T13:13:40.960+0000] {base_job.py:240} DEBUG - [heartbeat]\n[2023-09-13T13:13:46.444+0000] {base_job.py:240} DEBUG - [heartbeat]\n[2023-09-13T13:13:51.751+0000] {base_job.py:240} DEBUG - [heartbeat]\n[2023-09-13T13:13:57.084+0000] {base_job.py:240} DEBUG - [heartbeat]\n[2023-09-13T13:14:02.310+0000] {base_job.py:240} DEBUG - [heartbeat]\n[2023-09-13T13:14:07.535+0000] {base_job.py:240} DEBUG - [heartbeat]\n[2023-09-13T13:14:12.754+0000] {base_job.py:240} DEBUG - [heartbeat]\n[2023-09-13T13:14:17.967+0000] {base_job.py:240} DEBUG - [heartbeat]\n[2023-09-13T13:14:23.185+0000] {base_job.py:240} DEBUG - [heartbeat]\n[2023-09-13T13:14:28.406+0000] {base_job.py:240} DEBUG - [heartbeat]\n[2023-09-13T13:14:33.661+0000] {base_job.py:240} DEBUG - [heartbeat]\n[2023-09-13T13:14:38.883+0000] {base_job.py:240} DEBUG - [heartbeat]\n[2023-09-13T13:14:44.247+0000] {base_job.py:240} DEBUG - [heartbeat]```", + "user": "U05QL7LN2GH", + "ts": "1694610898.877169", + "blocks": [ + { + "type": "rich_text", + "block_id": "K1RCj", + "elements": [ + { + "type": "rich_text_preformatted", + "elements": [ + { + "type": "text", + "text": "Starting the process, got command: triggerer\nInitializing airflow.cfg.\nairflow.cfg initialization is done.\n[2023-09-13T13:11:46.620+0000] {settings.py:267} DEBUG - Setting up DB connection pool (PID 8)\n[2023-09-13T13:11:46.622+0000] {settings.py:372} DEBUG - settings.prepare_engine_args(): Using pool settings. pool_size=5, max_overflow=10, pool_recycle=570, pid=8\n[2023-09-13T13:11:46.742+0000] {cli_action_loggers.py:39} DEBUG - Adding to pre execution callback\n[2023-09-13T13:11:47.638+0000] {cli_action_loggers.py:65} DEBUG - Calling callbacks: []\n ____________ _____________\n ____ |__( )_________ __/__ /________ __\n____ /| |_ /__ ___/_ /_ __ /_ __ \\_ | /| / /\n___ ___ | / _ / _ __/ _ / / /_/ /_ |/ |/ /\n _/_/ |_/_/ /_/ /_/ /_/ \\____/____/|__/\n[2023-09-13T13:11:50.527+0000] {plugins_manager.py:300} DEBUG - Loading plugins\n[2023-09-13T13:11:50.580+0000] {plugins_manager.py:244} DEBUG - Loading plugins from directory: /home/airflow/gcs/plugins\n[2023-09-13T13:11:50.581+0000] {plugins_manager.py:224} DEBUG - Loading plugins from entrypoints\n[2023-09-13T13:11:50.587+0000] {plugins_manager.py:227} DEBUG - Importing entry_point plugin OpenLineagePlugin\n[2023-09-13T13:11:50.740+0000] {utils.py:430} WARNING - No module named 'boto3'\n[2023-09-13T13:11:50.743+0000] {utils.py:430} WARNING - No module named 'botocore'\n[2023-09-13T13:11:50.833+0000] {utils.py:430} WARNING - No module named 'airflow.providers.sftp'\n[2023-09-13T13:11:51.144+0000] {utils.py:430} WARNING - No module named 'big_query_insert_job_extractor'\n[2023-09-13T13:11:51.145+0000] {plugins_manager.py:237} ERROR - Failed to import plugin OpenLineagePlugin\nTraceback (most recent call last):\n File \"/opt/python3.8/lib/python3.8/site-packages/openlineage/airflow/utils.py\", line 427, in import_from_string\n module = importlib.import_module(module_path)\n File \"/opt/python3.8/lib/python3.8/importlib/__init__.py\", line 127, in import_module\n return _bootstrap._gcd_import(name[level:], package, level)\n File \"\", line 1014, in _gcd_import\n File \"\", line 991, in _find_and_load\n File \"\", line 973, in _find_and_load_unlocked\nModuleNotFoundError: No module named 'big_query_insert_job_extractor'\n\nThe above exception was the direct cause of the following exception:\n\nTraceback (most recent call last):\n File \"/opt/python3.8/lib/python3.8/site-packages/airflow/plugins_manager.py\", line 229, in load_entrypoint_plugins\n plugin_class = entry_point.load()\n File \"/opt/python3.8/lib/python3.8/site-packages/setuptools/_vendor/importlib_metadata/__init__.py\", line 194, in load\n module = import_module(match.group('module'))\n File \"/opt/python3.8/lib/python3.8/importlib/__init__.py\", line 127, in import_module\n return _bootstrap._gcd_import(name[level:], package, level)\n File \"\", line 1014, in _gcd_import\n File \"\", line 991, in _find_and_load\n File \"\", line 975, in _find_and_load_unlocked\n File \"\", line 671, in _load_unlocked\n File \"\", line 843, in exec_module\n File \"\", line 219, in _call_with_frames_removed\n File \"/opt/python3.8/lib/python3.8/site-packages/openlineage/airflow/plugin.py\", line 32, in \n from openlineage.airflow import listener\n File \"/opt/python3.8/lib/python3.8/site-packages/openlineage/airflow/listener.py\", line 75, in \n extractor_manager = ExtractorManager()\n File \"/opt/python3.8/lib/python3.8/site-packages/openlineage/airflow/extractors/manager.py\", line 16, in __init__\n self.task_to_extractor = Extractors()\n File \"/opt/python3.8/lib/python3.8/site-packages/openlineage/airflow/extractors/extractors.py\", line 122, in __init__\n extractor = import_from_string(extractor.strip())\n File \"/opt/python3.8/lib/python3.8/site-packages/openlineage/airflow/utils.py\", line 431, in import_from_string\n raise ImportError(f\"Failed to import {path}\") from e\nImportError: Failed to import big_query_insert_job_extractor.BigQueryInsertJobExtractor\n[2023-09-13T13:11:51.235+0000] {plugins_manager.py:227} DEBUG - Importing entry_point plugin composer_menu_plugin\n[2023-09-13T13:11:51.719+0000] {plugins_manager.py:316} DEBUG - Loading 1 plugin(s) took 1.14 seconds\n[2023-09-13T13:11:51.733+0000] {triggerer_job.py:101} INFO - Starting the triggerer\n[2023-09-13T13:11:51.734+0000] {selector_events.py:59} DEBUG - Using selector: EpollSelector\n[2023-09-13T13:11:56.118+0000] {base_job.py:240} DEBUG - [heartbeat]\n[2023-09-13T13:12:01.359+0000] {base_job.py:240} DEBUG - [heartbeat]\n[2023-09-13T13:12:06.665+0000] {base_job.py:240} DEBUG - [heartbeat]\n[2023-09-13T13:12:11.880+0000] {base_job.py:240} DEBUG - [heartbeat]\n[2023-09-13T13:12:17.098+0000] {base_job.py:240} DEBUG - [heartbeat]\n[2023-09-13T13:12:22.323+0000] {base_job.py:240} DEBUG - [heartbeat]\n[2023-09-13T13:12:27.597+0000] {base_job.py:240} DEBUG - [heartbeat]\n[2023-09-13T13:12:32.826+0000] {base_job.py:240} DEBUG - [heartbeat]\n[2023-09-13T13:12:38.049+0000] {base_job.py:240} DEBUG - [heartbeat]\n[2023-09-13T13:12:43.275+0000] {base_job.py:240} DEBUG - [heartbeat]\n[2023-09-13T13:12:48.509+0000] {base_job.py:240} DEBUG - [heartbeat]\n[2023-09-13T13:12:53.867+0000] {base_job.py:240} DEBUG - [heartbeat]\n[2023-09-13T13:12:59.087+0000] {base_job.py:240} DEBUG - [heartbeat]\n[2023-09-13T13:13:04.300+0000] {base_job.py:240} DEBUG - [heartbeat]\n[2023-09-13T13:13:09.539+0000] {base_job.py:240} DEBUG - [heartbeat]\n[2023-09-13T13:13:14.785+0000] {base_job.py:240} DEBUG - [heartbeat]\n[2023-09-13T13:13:20.007+0000] {base_job.py:240} DEBUG - [heartbeat]\n[2023-09-13T13:13:25.274+0000] {base_job.py:240} DEBUG - [heartbeat]\n[2023-09-13T13:13:30.510+0000] {base_job.py:240} DEBUG - [heartbeat]\n[2023-09-13T13:13:35.729+0000] {base_job.py:240} DEBUG - [heartbeat]\n[2023-09-13T13:13:40.960+0000] {base_job.py:240} DEBUG - [heartbeat]\n[2023-09-13T13:13:46.444+0000] {base_job.py:240} DEBUG - [heartbeat]\n[2023-09-13T13:13:51.751+0000] {base_job.py:240} DEBUG - [heartbeat]\n[2023-09-13T13:13:57.084+0000] {base_job.py:240} DEBUG - [heartbeat]\n[2023-09-13T13:14:02.310+0000] {base_job.py:240} DEBUG - [heartbeat]\n[2023-09-13T13:14:07.535+0000] {base_job.py:240} DEBUG - [heartbeat]\n[2023-09-13T13:14:12.754+0000] {base_job.py:240} DEBUG - [heartbeat]\n[2023-09-13T13:14:17.967+0000] {base_job.py:240} DEBUG - [heartbeat]\n[2023-09-13T13:14:23.185+0000] {base_job.py:240} DEBUG - [heartbeat]\n[2023-09-13T13:14:28.406+0000] {base_job.py:240} DEBUG - [heartbeat]\n[2023-09-13T13:14:33.661+0000] {base_job.py:240} DEBUG - [heartbeat]\n[2023-09-13T13:14:38.883+0000] {base_job.py:240} DEBUG - [heartbeat]\n[2023-09-13T13:14:44.247+0000] {base_job.py:240} DEBUG - [heartbeat]" + } + ], + "border": 0 + } + ] + } + ], + "team": "T01CWUYP5AR", + "thread_ts": "1694545905.974339", + "parent_user_id": "U05QL7LN2GH" + }, + { + "client_msg_id": "233df8a1-29ac-45fc-9e9b-dcb1926b9628", + "type": "message", + "text": "still the same error in the triggerer pod", + "user": "U05QL7LN2GH", + "ts": "1694610910.432769", + "blocks": [ + { + "type": "rich_text", + "block_id": "6w/Pi", + "elements": [ + { + "type": "rich_text_section", + "elements": [ + { + "type": "text", + "text": "still the same error in the triggerer pod" + } + ] + } + ] + } + ], + "team": "T01CWUYP5AR", + "thread_ts": "1694545905.974339", + "parent_user_id": "U05QL7LN2GH" + }, + { + "type": "message", + "text": "have changed the dags folder where i have added the init file as you suggested and then have updated the OPENLINEAGE_EXTRACTORS to big_query_insert_job_extractor.BigQueryInsertJobExtractor…still the same thing", + "files": [ + { + "id": "F05S5NP9N7M", + "created": 1694610929, + "timestamp": 1694610929, + "name": "Screenshot 2023-09-13 at 6.45.26 PM.png", + "title": "Screenshot 2023-09-13 at 6.45.26 PM.png", + "mimetype": "image/png", + "filetype": "png", + "pretty_type": "PNG", + "user": "U05QL7LN2GH", + "user_team": "T01CWUYP5AR", + "editable": false, + "size": 997510, + "mode": "hosted", + "is_external": false, + "external_type": "", + "is_public": true, + "public_url_shared": false, + "display_as_bot": false, + "username": "", + "url_private": "https://files.slack.com/files-pri/T01CWUYP5AR-F05S5NP9N7M/screenshot_2023-09-13_at_6.45.26_pm.png", + "url_private_download": "https://files.slack.com/files-pri/T01CWUYP5AR-F05S5NP9N7M/download/screenshot_2023-09-13_at_6.45.26_pm.png", + "media_display_type": "unknown", + "thumb_64": "https://files.slack.com/files-tmb/T01CWUYP5AR-F05S5NP9N7M-a5b84be03b/screenshot_2023-09-13_at_6.45.26_pm_64.png", + "thumb_80": "https://files.slack.com/files-tmb/T01CWUYP5AR-F05S5NP9N7M-a5b84be03b/screenshot_2023-09-13_at_6.45.26_pm_80.png", + "thumb_360": "https://files.slack.com/files-tmb/T01CWUYP5AR-F05S5NP9N7M-a5b84be03b/screenshot_2023-09-13_at_6.45.26_pm_360.png", + "thumb_360_w": 360, + "thumb_360_h": 126, + "thumb_480": "https://files.slack.com/files-tmb/T01CWUYP5AR-F05S5NP9N7M-a5b84be03b/screenshot_2023-09-13_at_6.45.26_pm_480.png", + "thumb_480_w": 480, + "thumb_480_h": 169, + "thumb_160": "https://files.slack.com/files-tmb/T01CWUYP5AR-F05S5NP9N7M-a5b84be03b/screenshot_2023-09-13_at_6.45.26_pm_160.png", + "thumb_720": "https://files.slack.com/files-tmb/T01CWUYP5AR-F05S5NP9N7M-a5b84be03b/screenshot_2023-09-13_at_6.45.26_pm_720.png", + "thumb_720_w": 720, + "thumb_720_h": 253, + "thumb_800": "https://files.slack.com/files-tmb/T01CWUYP5AR-F05S5NP9N7M-a5b84be03b/screenshot_2023-09-13_at_6.45.26_pm_800.png", + "thumb_800_w": 800, + "thumb_800_h": 281, + "thumb_960": "https://files.slack.com/files-tmb/T01CWUYP5AR-F05S5NP9N7M-a5b84be03b/screenshot_2023-09-13_at_6.45.26_pm_960.png", + "thumb_960_w": 960, + "thumb_960_h": 337, + "thumb_1024": "https://files.slack.com/files-tmb/T01CWUYP5AR-F05S5NP9N7M-a5b84be03b/screenshot_2023-09-13_at_6.45.26_pm_1024.png", + "thumb_1024_w": 1024, + "thumb_1024_h": 360, + "original_w": 2892, + "original_h": 1016, + "thumb_tiny": "AwAQADDRx/nNA4oBpaAFzRSd+tHPrQAtFJ+NJigD/9k=", + "permalink": "https://openlineage.slack.com/files/U05QL7LN2GH/F05S5NP9N7M/screenshot_2023-09-13_at_6.45.26_pm.png", + "permalink_public": "https://slack-files.com/T01CWUYP5AR-F05S5NP9N7M-436a8b87bc", + "is_starred": false, + "has_rich_preview": false, + "file_access": "visible" + } + ], + "upload": false, + "user": "U05QL7LN2GH", + "display_as_bot": false, + "ts": "1694610983.761409", + "blocks": [ + { + "type": "rich_text", + "block_id": "CFHx4", + "elements": [ + { + "type": "rich_text_section", + "elements": [ + { + "type": "text", + "text": "have changed the dags folder where i have added the init file as you suggested and then have updated the OPENLINEAGE_EXTRACTORS to big_query_insert_job_extractor.BigQueryInsertJobExtractor…still the same thing" + } + ] + } + ] + } + ], + "client_msg_id": "8031872d-2c92-4071-98a5-8eef81bbdf58", + "thread_ts": "1694545905.974339", + "parent_user_id": "U05QL7LN2GH" + }, + { + "client_msg_id": "73d99e91-e4ad-4a9a-90ce-50cf16106e0b", + "type": "message", + "text": "> still the same error in the triggerer pod\nit won't change, we're not trying to fix the triggerer import but worker, and should look only at worker pod at this point", + "user": "U01RA9B5GG2", + "ts": "1694612187.192779", + "blocks": [ + { + "type": "rich_text", + "block_id": "UEEwt", + "elements": [ + { + "type": "rich_text_quote", + "elements": [ + { + "type": "text", + "text": "still the same error in the triggerer pod" + } + ] + }, + { + "type": "rich_text_section", + "elements": [ + { + "type": "text", + "text": "it won't change, we're not trying to fix the triggerer import but worker, and should look only at worker pod at this point" + } + ] + } + ] + } + ], + "team": "T01CWUYP5AR", + "edited": { + "user": "U01RA9B5GG2", + "ts": "1694612229.000000" + }, + "thread_ts": "1694545905.974339", + "parent_user_id": "U05QL7LN2GH" + }, + { + "client_msg_id": "c930bd41-af54-463f-b011-37125a0ec49c", + "type": "message", + "text": "```extractor for <class 'airflow.providers.google.cloud.operators.bigquery.BigQueryInsertJobOperator'> is <class 'big_query_insert_job_extractor.BigQueryInsertJobExtractor'\n\nUsing extractor BigQueryInsertJobExtractor task_type=BigQueryInsertJobOperator airflow_dag_id=data_analytics_dag task_id=join_bq_datasets.bq_join_holidays_weather_data_2021 airflow_run_id=manual__2023-09-13T13:24:08.946947+00:00 \n\nfatal: not a git repository (or any parent up to mount point /home/airflow)\nStopping at filesystem boundary (GIT_DISCOVERY_ACROSS_FILESYSTEM not set).\nfatal: not a git repository (or any parent up to mount point /home/airflow)\nStopping at filesystem boundary (GIT_DISCOVERY_ACROSS_FILESYSTEM not set).```", + "user": "U05QL7LN2GH", + "ts": "1694612614.195119", + "blocks": [ + { + "type": "rich_text", + "block_id": "xTXcg", + "elements": [ + { + "type": "rich_text_preformatted", + "elements": [ + { + "type": "text", + "text": "extractor for is is ", + "style": { + "code": true + } + } + ] + } + ] + } + ], + "team": "T01CWUYP5AR", + "thread_ts": "1694545905.974339", + "parent_user_id": "U05QL7LN2GH" + }, + { + "client_msg_id": "707e745e-34c7-41e1-ad73-199ea31abe09", + "type": "message", + "text": "no `__init__.py` in base `dags` folder", + "user": "U02S6F54MAB", + "ts": "1694614562.336989", + "blocks": [ + { + "type": "rich_text", + "block_id": "X8CD7", + "elements": [ + { + "type": "rich_text_section", + "elements": [ + { + "type": "text", + "text": "no " + }, + { + "type": "text", + "text": "__init__.py", + "style": { + "code": true + } + }, + { + "type": "text", + "text": " in base " + }, + { + "type": "text", + "text": "dags", + "style": { + "code": true + } + }, + { + "type": "text", + "text": " folder" + } + ] + } + ] + } + ], + "team": "T01CWUYP5AR", + "thread_ts": "1694545905.974339", + "parent_user_id": "U05QL7LN2GH" + }, + { + "client_msg_id": "d83ea232-7514-4559-b8aa-afc6e49eb685", + "type": "message", + "text": "I also checked that triggerer pod indeed has no gcsfuse set up, tbh no idea why, maybe some kind of optimization\nthe only effect is that when loading plugins in triggerer it throws some errors in logs, we don’t do anything at the moment there", + "user": "U02S6F54MAB", + "ts": "1694614622.751639", + "blocks": [ + { + "type": "rich_text", + "block_id": "kGlCl", + "elements": [ + { + "type": "rich_text_section", + "elements": [ + { + "type": "text", + "text": "I also checked that triggerer pod indeed has no gcsfuse set up, tbh no idea why, maybe some kind of optimization\nthe only effect is that when loading plugins in triggerer it throws some errors in logs, we don’t do anything at the moment there" + } + ] + } + ] + } + ], + "team": "T01CWUYP5AR", + "thread_ts": "1694545905.974339", + "parent_user_id": "U05QL7LN2GH" + }, + { + "client_msg_id": "4a20640a-4af5-4964-98cf-1fcfa16885a9", + "type": "message", + "text": "okk…got it <@U02S6F54MAB>…so the init at the top level of dags is as well not reqd, got it. Just one more doubt, there is a requirement where i want to change the operators property in the extractor inside the extract function, will that be taken into account and the operator’s execute be called with the property that i have populated in my extractor?", + "user": "U05QL7LN2GH", + "ts": "1694614766.540349", + "blocks": [ + { + "type": "rich_text", + "block_id": "u4Kyz", + "elements": [ + { + "type": "rich_text_section", + "elements": [ + { + "type": "text", + "text": "okk…got it " + }, + { + "type": "user", + "user_id": "U02S6F54MAB" + }, + { + "type": "text", + "text": "…so the init at the top level of dags is as well not reqd, got it. Just one more doubt, there is a requirement where i want to change the operators property in the extractor inside the extract function, will that be taken into account and the operator’s execute be called with the property that i have populated in my extractor?" + } + ] + } + ] + } + ], + "team": "T01CWUYP5AR", + "edited": { + "user": "U05QL7LN2GH", + "ts": "1694614794.000000" + }, + "thread_ts": "1694545905.974339", + "parent_user_id": "U05QL7LN2GH" + }, + { + "client_msg_id": "5d797ae3-ddd8-4cc4-9d0c-05da684c2c4c", + "type": "message", + "text": "for example i want to add a custom job_id to the BigQueryInsertJobOperator, so wheneerv someone uses the BigQueryInsertJobOperator operator i want to intercept that and add this job_id property to the operator…will that work?", + "user": "U05QL7LN2GH", + "ts": "1694614888.624699", + "blocks": [ + { + "type": "rich_text", + "block_id": "Qo9jq", + "elements": [ + { + "type": "rich_text_section", + "elements": [ + { + "type": "text", + "text": "for example i want to add a custom job_id to the BigQueryInsertJobOperator, so wheneerv someone uses the BigQueryInsertJobOperator operator i want to intercept that and add this job_id property to the operator…will that work?" + } + ] + } + ] + } + ], + "team": "T01CWUYP5AR", + "thread_ts": "1694545905.974339", + "parent_user_id": "U05QL7LN2GH" + }, + { + "client_msg_id": "63c25b13-9833-4adc-a4f3-b5429d113d7b", + "type": "message", + "text": "I’m not sure if using OL for such thing is best choice. Wouldn’t it be better to subclass the operator?", + "user": "U02S6F54MAB", + "ts": "1694615086.037269", + "blocks": [ + { + "type": "rich_text", + "block_id": "9kpzj", + "elements": [ + { + "type": "rich_text_section", + "elements": [ + { + "type": "text", + "text": "I’m not sure if using OL for such thing is best choice. Wouldn’t it be better to subclass the operator?" + } + ] + } + ] + } + ], + "team": "T01CWUYP5AR", + "thread_ts": "1694545905.974339", + "parent_user_id": "U05QL7LN2GH" + }, + { + "client_msg_id": "599fa46f-72bb-4aec-8130-ed0ef0ccd585", + "type": "message", + "text": "but the answer is: it dependes on the airflow version, in 2.3+ I’m pretty sure the changed property stays in execute method", + "user": "U02S6F54MAB", + "ts": "1694615137.622649", + "blocks": [ + { + "type": "rich_text", + "block_id": "lDV/q", + "elements": [ + { + "type": "rich_text_section", + "elements": [ + { + "type": "text", + "text": "but the answer is: it dependes on the airflow version, in 2.3+ I’m pretty sure the changed property stays in execute method" + } + ] + } + ] + } + ], + "team": "T01CWUYP5AR", + "thread_ts": "1694545905.974339", + "parent_user_id": "U05QL7LN2GH" + }, + { + "client_msg_id": "c272da2c-cb40-4bbc-884a-23d96e678d05", + "type": "message", + "text": "yeah ideally that is how we should have done this but the problem is our client is having around 1000+ Dag’s in different google cloud projects, which are owned by multiple teams…so they are not willing to change anything in their dag. Thankfully they are using airflow 2.4.3", + "user": "U05QL7LN2GH", + "ts": "1694615269.140809", + "blocks": [ + { + "type": "rich_text", + "block_id": "RPkDF", + "elements": [ + { + "type": "rich_text_section", + "elements": [ + { + "type": "text", + "text": "yeah ideally that is how we should have done this but the problem is our client is having around 1000+ Dag’s in different google cloud projects, which are owned by multiple teams…so they are not willing to change anything in their dag. Thankfully they are using airflow 2.4.3" + } + ] + } + ] + } + ], + "team": "T01CWUYP5AR", + "thread_ts": "1694545905.974339", + "parent_user_id": "U05QL7LN2GH" + }, + { + "client_msg_id": "258df1ca-7e1a-4a4c-8b80-60ad502d1faf", + "type": "message", + "text": "task_policy might be better tool for that: ", + "user": "U01RA9B5GG2", + "ts": "1694615475.647459", + "blocks": [ + { + "type": "rich_text", + "block_id": "76umU", + "elements": [ + { + "type": "rich_text_section", + "elements": [ + { + "type": "text", + "text": "task_policy might be better tool for that: " + }, + { + "type": "link", + "url": "https://airflow.apache.org/docs/apache-airflow/2.6.0/administration-and-deployment/cluster-policies.html" + } + ] + } + ] + } + ], + "team": "T01CWUYP5AR", + "thread_ts": "1694545905.974339", + "parent_user_id": "U05QL7LN2GH", + "reactions": [ + { + "name": "heavy_plus_sign", + "users": [ + "U02S6F54MAB" + ], + "count": 1 + } + ] + }, + { + "client_msg_id": "5b7efe5d-f026-40af-9b28-721f4f06a322", + "type": "message", + "text": "btw I double-checked - execute method is in different process so this would not change task’s attribute there", + "user": "U02S6F54MAB", + "ts": "1694615730.010879", + "blocks": [ + { + "type": "rich_text", + "block_id": "dseTu", + "elements": [ + { + "type": "rich_text_section", + "elements": [ + { + "type": "text", + "text": "btw I double-checked - execute method is in different process so this would not change task’s attribute there" + } + ] + } + ] + } + ], + "team": "T01CWUYP5AR", + "thread_ts": "1694545905.974339", + "parent_user_id": "U05QL7LN2GH" + }, + { + "client_msg_id": "0fc11194-8888-436b-a4c7-ce13fe6392f5", + "type": "message", + "text": "<@U02S6F54MAB> any idea how can we achieve this one. ---> ", + "user": "U05QL7LN2GH", + "ts": "1694849569.693149", + "blocks": [ + { + "type": "rich_text", + "block_id": "wzGyQ", + "elements": [ + { + "type": "rich_text_section", + "elements": [ + { + "type": "user", + "user_id": "U02S6F54MAB" + }, + { + "type": "text", + "text": " any idea how can we achieve this one. ---> " + }, + { + "type": "link", + "url": "https://openlineage.slack.com/archives/C01CK9T7HKR/p1694849427228709" + } + ] + } + ] + } + ], + "team": "T01CWUYP5AR", + "attachments": [ + { + "from_url": "https://openlineage.slack.com/archives/C01CK9T7HKR/p1694849427228709", + "ts": "1694849427.228709", + "author_id": "U05QL7LN2GH", + "channel_id": "C01CK9T7HKR", + "channel_team": "T01CWUYP5AR", + "is_msg_unfurl": true, + "message_blocks": [ + { + "team": "T01CWUYP5AR", + "channel": "C01CK9T7HKR", + "ts": "1694849427.228709", + "message": { + "blocks": [ + { + "type": "rich_text", + "block_id": "AHZek", + "elements": [ + { + "type": "rich_text_section", + "elements": [ + { + "type": "broadcast", + "range": "here" + }, + { + "type": "text", + "text": " we have dataproc operator getting called from a dag which submits a spark job, we wanted to maintain that continuity of parent job in the spark job so according to the documentation we can acheive that by using a macro called lineage_run_id that requires task and task_instance as the parameters. The problem we are facing is that our client’s have 1000's of dags, so asking them to change this everywhere it is used is not feasible, so we thought of using the task_policy feature in the airflow…but the problem is that task_policy gives you access to only the task/operator, but we don’t have the access to the task instance..that is required as a parameter to the lineage_run_id function. Can anyone kindly help us on how should we go about this one\n" + } + ] + }, + { + "type": "rich_text_preformatted", + "elements": [ + { + "type": "text", + "text": "t1 = DataProcPySparkOperator(\n task_id=job_name,\n #required pyspark configuration,\n job_name=job_name,\n dataproc_pyspark_properties={\n 'spark.driver.extraJavaOptions':\n f\"-javaagent:{jar}={os.environ.get('OPENLINEAGE_URL')}/api/v1/namespaces/{os.getenv('OPENLINEAGE_NAMESPACE', 'default')}/jobs/{job_name}/runs/{{{{macros.OpenLineagePlugin.lineage_run_id(task, task_instance)}}}}?api_key={os.environ.get('OPENLINEAGE_API_KEY')}\"\n dag=dag)" + } + ], + "border": 0 + } + ] + } + ] + } + } + ], + "id": 1, + "original_url": "https://openlineage.slack.com/archives/C01CK9T7HKR/p1694849427228709", + "fallback": "[September 16th, 2023 12:30 AM] jeevan: we have dataproc operator getting called from a dag which submits a spark job, we wanted to maintain that continuity of parent job in the spark job so according to the documentation we can acheive that by using a macro called lineage_run_id that requires task and task_instance as the parameters. The problem we are facing is that our client’s have 1000's of dags, so asking them to change this everywhere it is used is not feasible, so we thought of using the task_policy feature in the airflow…but the problem is that task_policy gives you access to only the task/operator, but we don’t have the access to the task instance..that is required as a parameter to the lineage_run_id function. Can anyone kindly help us on how should we go about this one\n```t1 = DataProcPySparkOperator(\n task_id=job_name,\n #required pyspark configuration,\n job_name=job_name,\n dataproc_pyspark_properties={\n 'spark.driver.extraJavaOptions':\n f\"-javaagent:{jar}={os.environ.get('OPENLINEAGE_URL')}/api/v1/namespaces/{os.getenv('OPENLINEAGE_NAMESPACE', 'default')}/jobs/{job_name}/runs/{{{{macros.OpenLineagePlugin.lineage_run_id(task, task_instance)}}}}?api_key={os.environ.get('OPENLINEAGE_API_KEY')}\"\n dag=dag)```", + "text": " we have dataproc operator getting called from a dag which submits a spark job, we wanted to maintain that continuity of parent job in the spark job so according to the documentation we can acheive that by using a macro called lineage_run_id that requires task and task_instance as the parameters. The problem we are facing is that our client’s have 1000's of dags, so asking them to change this everywhere it is used is not feasible, so we thought of using the task_policy feature in the airflow…but the problem is that task_policy gives you access to only the task/operator, but we don’t have the access to the task instance..that is required as a parameter to the lineage_run_id function. Can anyone kindly help us on how should we go about this one\n```t1 = DataProcPySparkOperator(\n task_id=job_name,\n #required pyspark configuration,\n job_name=job_name,\n dataproc_pyspark_properties={\n 'spark.driver.extraJavaOptions':\n f\"-javaagent:{jar}={os.environ.get('OPENLINEAGE_URL')}/api/v1/namespaces/{os.getenv('OPENLINEAGE_NAMESPACE', 'default')}/jobs/{job_name}/runs/{{{{macros.OpenLineagePlugin.lineage_run_id(task, task_instance)}}}}?api_key={os.environ.get('OPENLINEAGE_API_KEY')}\"\n dag=dag)```", + "author_name": "Guntaka Jeevan Paul", + "author_link": "https://openlineage.slack.com/team/U05QL7LN2GH", + "author_icon": "https://avatars.slack-edge.com/2023-08-30/5820708969061_5ef61b937b8b8e9b2d6e_48.png", + "author_subname": "Guntaka Jeevan Paul", + "mrkdwn_in": [ + "text" + ], + "footer": "Slack Conversation" + } + ], + "thread_ts": "1694545905.974339", + "parent_user_id": "U05QL7LN2GH" + } + ] + }, + { + "client_msg_id": "80f8a117-68eb-4bc7-9352-9d5254dab39c", + "type": "message", + "text": "This particular code in docker-compose exits with code 1 because it is unable to find wait-for-it.sh, file in the container. I have checked the mounting path from the local machine, It is correct and the path on the container for Marquez is also correct i.e. /usr/src/app but it is unable to mount the wait-for-it.sh. Does anyone know why is this? This code exists in the open lineage repository as well \n```# Marquez as an OpenLineage Client\n api:\n image: marquezproject/marquez\n container_name: marquez-api\n ports:\n - \"5000:5000\"\n - \"5001:5001\"\n volumes:\n - ./docker/wait-for-it.sh:/usr/src/app/wait-for-it.sh\n links:\n - \"db:postgres\"\n depends_on:\n - db\n entrypoint: [ \"./wait-for-it.sh\", \"db:5432\", \"--\", \"./entrypoint.sh\" ]```", + "user": "U05QNRSQW1E", + "ts": "1694520846.519609", + "blocks": [ + { + "type": "rich_text", + "block_id": "DEp/x", + "elements": [ + { + "type": "rich_text_section", + "elements": [ + { + "type": "text", + "text": "This particular code in docker-compose exits with code 1 because it is unable to find wait-for-it.sh, file in the container. I have checked the mounting path from the local machine, It is correct and the path on the container for Marquez is also correct i.e. /usr/src/app but it is unable to mount the wait-for-it.sh. Does anyone know why is this? This code exists in the open lineage repository as well " + }, + { + "type": "link", + "url": "https://github.com/OpenLineage/OpenLineage/blob/main/integration/spark/docker-compose.yml" + }, + { + "type": "text", + "text": "\n" + } + ] + }, + { + "type": "rich_text_preformatted", + "elements": [ + { + "type": "text", + "text": "# Marquez as an OpenLineage Client\n api:\n image: marquezproject/marquez\n container_name: marquez-api\n ports:\n - \"5000:5000\"\n - \"5001:5001\"\n volumes:\n - ./docker/wait-for-it.sh:/usr/src/app/wait-for-it.sh\n links:\n - \"db:postgres\"\n depends_on:\n - db\n entrypoint: [ \"./wait-for-it.sh\", \"db:5432\", \"--\", \"./entrypoint.sh\" ]" + } + ], + "border": 0 + } + ] + } + ], + "team": "T01CWUYP5AR", + "attachments": [ + { + "id": 1, + "footer_icon": "https://slack.github.com/static/img/favicon-neutral.png", + "color": "24292f", + "bot_id": "B01VA0FB340", + "app_unfurl_url": "https://github.com/OpenLineage/OpenLineage/blob/main/integration/spark/docker-compose.yml", + "is_app_unfurl": true, + "app_id": "A01BP7R4KNY", + "fallback": "", + "text": "```\nversion: \"3.7\"\nservices:\n notebook:\n image: jupyter/pyspark-notebook:spark-3.1.2\n ports:\n - \"8888:8888\"\n volumes:\n - ./docker/notebooks:/home/jovyan/notebooks\n - ./build:/home/jovyan/openlineage\n links:\n - \"api:marquez\"\n depends_on:\n - api\n\n# Marquez as an OpenLineage Client\n api:\n image: marquezproject/marquez\n container_name: marquez-api\n ports:\n - \"5000:5000\"\n - \"5001:5001\"\n volumes:\n - ./docker/wait-for-it.sh:/usr/src/app/wait-for-it.sh\n links:\n - \"db:postgres\"\n depends_on:\n - db\n entrypoint: [ \"./wait-for-it.sh\", \"db:5432\", \"--\", \"./entrypoint.sh\" ]\n\n db:\n image: postgres:12.1\n container_name: marquez-db\n ports:\n - \"5432:5432\"\n environment:\n - POSTGRES_USER=postgres\n - POSTGRES_PASSWORD=password\n - MARQUEZ_DB=marquez\n - MARQUEZ_USER=marquez\n - MARQUEZ_PASSWORD=marquez\n volumes:\n - ./docker/init-db.sh:/docker-entrypoint-initdb.d/init-db.sh\n # Enables SQL statement logging (see: )\n # command: [\"postgres\", \"-c\", \"log_statement=all\"]\n```", + "title": "", + "footer": "", + "mrkdwn_in": [ + "text" + ] + } + ], + "thread_ts": "1694520846.519609", + "reply_count": 2, + "reply_users_count": 2, + "latest_reply": "1694529521.989119", + "reply_users": [ + "U05QNRSQW1E", + "U01RA9B5GG2" + ], + "is_locked": false, + "subscribed": false, + "replies": [ + { + "type": "message", + "text": "This is the error message:", + "files": [ + { + "id": "F05RZ520LH0", + "created": 1694520912, + "timestamp": 1694520912, + "name": "image.png", + "title": "image.png", + "mimetype": "image/png", + "filetype": "png", + "pretty_type": "PNG", + "user": "U05QNRSQW1E", + "user_team": "T01CWUYP5AR", + "editable": false, + "size": 38678, + "mode": "hosted", + "is_external": false, + "external_type": "", + "is_public": true, + "public_url_shared": false, + "display_as_bot": false, + "username": "", + "url_private": "https://files.slack.com/files-pri/T01CWUYP5AR-F05RZ520LH0/image.png", + "url_private_download": "https://files.slack.com/files-pri/T01CWUYP5AR-F05RZ520LH0/download/image.png", + "media_display_type": "unknown", + "thumb_64": "https://files.slack.com/files-tmb/T01CWUYP5AR-F05RZ520LH0-31293b6eef/image_64.png", + "thumb_80": "https://files.slack.com/files-tmb/T01CWUYP5AR-F05RZ520LH0-31293b6eef/image_80.png", + "thumb_360": "https://files.slack.com/files-tmb/T01CWUYP5AR-F05RZ520LH0-31293b6eef/image_360.png", + "thumb_360_w": 360, + "thumb_360_h": 69, + "thumb_480": "https://files.slack.com/files-tmb/T01CWUYP5AR-F05RZ520LH0-31293b6eef/image_480.png", + "thumb_480_w": 480, + "thumb_480_h": 92, + "thumb_160": "https://files.slack.com/files-tmb/T01CWUYP5AR-F05RZ520LH0-31293b6eef/image_160.png", + "thumb_720": "https://files.slack.com/files-tmb/T01CWUYP5AR-F05RZ520LH0-31293b6eef/image_720.png", + "thumb_720_w": 720, + "thumb_720_h": 138, + "thumb_800": "https://files.slack.com/files-tmb/T01CWUYP5AR-F05RZ520LH0-31293b6eef/image_800.png", + "thumb_800_w": 800, + "thumb_800_h": 153, + "thumb_960": "https://files.slack.com/files-tmb/T01CWUYP5AR-F05RZ520LH0-31293b6eef/image_960.png", + "thumb_960_w": 960, + "thumb_960_h": 183, + "thumb_1024": "https://files.slack.com/files-tmb/T01CWUYP5AR-F05RZ520LH0-31293b6eef/image_1024.png", + "thumb_1024_w": 1024, + "thumb_1024_h": 196, + "original_w": 1162, + "original_h": 222, + "thumb_tiny": "AwAJADCjnAo6jpTaevQUANxnt+tJgen608dKU0AREe4pKf3oboaAP//Z", + "permalink": "https://openlineage.slack.com/files/U05QNRSQW1E/F05RZ520LH0/image.png", + "permalink_public": "https://slack-files.com/T01CWUYP5AR-F05RZ520LH0-0af16fdefc", + "is_starred": false, + "has_rich_preview": false, + "file_access": "visible" + } + ], + "upload": false, + "user": "U05QNRSQW1E", + "display_as_bot": false, + "ts": "1694520919.480389", + "blocks": [ + { + "type": "rich_text", + "block_id": "OE+KD", + "elements": [ + { + "type": "rich_text_section", + "elements": [ + { + "type": "text", + "text": "This is the error message:" + } + ] + } + ] + } + ], + "client_msg_id": "fad44818-d26b-4144-857a-27c72de151f9", + "thread_ts": "1694520846.519609", + "parent_user_id": "U05QNRSQW1E" + }, + { + "client_msg_id": "b6974478-e3b6-4ce1-9e1e-dd32299bde91", + "type": "message", + "text": "no permissions?", + "user": "U01RA9B5GG2", + "ts": "1694529521.989119", + "blocks": [ + { + "type": "rich_text", + "block_id": "7Uh5u", + "elements": [ + { + "type": "rich_text_section", + "elements": [ + { + "type": "text", + "text": "no permissions?" + } + ] + } + ] + } + ], + "team": "T01CWUYP5AR", + "thread_ts": "1694520846.519609", + "parent_user_id": "U05QNRSQW1E" + } + ] + }, + { + "type": "message", + "subtype": "thread_broadcast", + "text": "Opened a PR for this here: ", + "user": "U04AZ7992SU", + "ts": "1694468262.274069", + "thread_ts": "1694466446.599329", + "root": { + "client_msg_id": "375d0fac-c7ba-4741-bf3b-4194cf48df0e", + "type": "message", + "text": "I’m seeing some odd behavior with my http transport when upgrading airflow/openlineage-airflow from 2.3.2 -> 2.6.3 and 0.24.0 -> 0.28.0. Previously I had a config like this that let me provide my own auth tokens. However, after upgrading I’m getting a 401 from the endpoint and further debugging seems to reveal that we’re not using the token provided in my TokenProvider. Does anyone know if something changed between these versions that could be causing this? (more details in :thread: )\n```transport:\n type: http\n url: \n auth:\n type: some.fully.qualified.classpath```", + "user": "U04AZ7992SU", + "ts": "1694466446.599329", + "blocks": [ + { + "type": "rich_text", + "block_id": "zNf8V", + "elements": [ + { + "type": "rich_text_section", + "elements": [ + { + "type": "text", + "text": "I’m seeing some odd behavior with my http transport when upgrading airflow/openlineage-airflow from 2.3.2 -> 2.6.3 and 0.24.0 -> 0.28.0. Previously I had a config like this that let me provide my own auth tokens. However, after upgrading I’m getting a 401 from the endpoint and further debugging seems to reveal that we’re not using the token provided in my TokenProvider. Does anyone know if something changed between these versions that could be causing this? (more details in " + }, + { + "type": "emoji", + "name": "thread", + "unicode": "1f9f5" + }, + { + "type": "text", + "text": " )\n" + } + ] + }, + { + "type": "rich_text_preformatted", + "elements": [ + { + "type": "text", + "text": "transport:\n type: http\n url: " + }, + { + "type": "link", + "url": "https://my.fake-marquez-endpoint.com" + }, + { + "type": "text", + "text": "\n auth:\n type: some.fully.qualified.classpath" + } + ], + "border": 0 + } + ] + } + ], + "team": "T01CWUYP5AR", + "edited": { + "user": "U04AZ7992SU", + "ts": "1694466508.000000" + }, + "thread_ts": "1694466446.599329", + "reply_count": 4, + "reply_users_count": 1, + "latest_reply": "1694468262.274069", + "reply_users": [ + "U04AZ7992SU" + ], + "is_locked": false, + "subscribed": false + }, + "blocks": [ + { + "type": "rich_text", + "block_id": "Sua8X", + "elements": [ + { + "type": "rich_text_section", + "elements": [ + { + "type": "text", + "text": "Opened a PR for this here: " + }, + { + "type": "link", + "url": "https://github.com/OpenLineage/OpenLineage/pull/2100" + } + ] + } + ] + } + ], + "attachments": [ + { + "id": 1, + "footer_icon": "https://slack.github.com/static/img/favicon-neutral.png", + "ts": 1694468121, + "color": "36a64f", + "bot_id": "B01VA0FB340", + "app_unfurl_url": "https://github.com/OpenLineage/OpenLineage/pull/2100", + "is_app_unfurl": true, + "app_id": "A01BP7R4KNY", + "fallback": "#2100 python: fix custom http transport TokenProvider", + "text": "*Problem*\n\nLooks like in we introduced a bug where we would always use the base `TokenProvider` class instead of a given custom token provider, even if the validation conditions passed.\n\n*Solution*\n\nRevert this to instantiate the subclass\n\n☐ Your change modifies the OpenLineage model\n☐ Your change modifies one or more OpenLineage \n\nIf you're contributing a new integration, please specify the scope of the integration and how/where it has been tested (e.g., Apache Spark integration supports `S3` and `GCS` filesystem operations, tested with AWS EMR).\n\n*One-line summary:*\n*Checklist*\n\n☑︎ You've your work\n☑︎ Your pull request title follows our \n☐ Your changes are accompanied by tests (_if relevant_)\n☑︎ Your change contains a and is self-contained\n☐ You've updated any relevant documentation (_if relevant_)\n☐ Your comment includes a one-liner for the changelog about the specific purpose of the change (_if necessary_)\n☐ You've versioned the core OpenLineage model or facets according to (_if relevant_)\n☐ You've added a to source files (_if relevant_)\n\n* * *\n\nSPDX-License-Identifier: Apache-2.0 \nCopyright 2018-2023 contributors to the OpenLineage project", + "title": "#2100 python: fix custom http transport TokenProvider", + "title_link": "https://github.com/OpenLineage/OpenLineage/pull/2100", + "footer": "", + "fields": [ + { + "value": "client/python", + "title": "Labels", + "short": true + }, + { + "value": "1", + "title": "Comments", + "short": true + } + ], + "mrkdwn_in": [ + "text" + ] + } + ], + "client_msg_id": "880a3f24-c44a-4818-ad1c-2eef97fec037", + "reactions": [ + { + "name": "heart", + "users": [ + "U01DCLP0GU9" + ], + "count": 1 + } + ] + }, + { + "client_msg_id": "375d0fac-c7ba-4741-bf3b-4194cf48df0e", + "type": "message", + "text": "I’m seeing some odd behavior with my http transport when upgrading airflow/openlineage-airflow from 2.3.2 -> 2.6.3 and 0.24.0 -> 0.28.0. Previously I had a config like this that let me provide my own auth tokens. However, after upgrading I’m getting a 401 from the endpoint and further debugging seems to reveal that we’re not using the token provided in my TokenProvider. Does anyone know if something changed between these versions that could be causing this? (more details in :thread: )\n```transport:\n type: http\n url: \n auth:\n type: some.fully.qualified.classpath```", + "user": "U04AZ7992SU", + "ts": "1694466446.599329", + "blocks": [ + { + "type": "rich_text", + "block_id": "zNf8V", + "elements": [ + { + "type": "rich_text_section", + "elements": [ + { + "type": "text", + "text": "I’m seeing some odd behavior with my http transport when upgrading airflow/openlineage-airflow from 2.3.2 -> 2.6.3 and 0.24.0 -> 0.28.0. Previously I had a config like this that let me provide my own auth tokens. However, after upgrading I’m getting a 401 from the endpoint and further debugging seems to reveal that we’re not using the token provided in my TokenProvider. Does anyone know if something changed between these versions that could be causing this? (more details in " + }, + { + "type": "emoji", + "name": "thread", + "unicode": "1f9f5" + }, + { + "type": "text", + "text": " )\n" + } + ] + }, + { + "type": "rich_text_preformatted", + "elements": [ + { + "type": "text", + "text": "transport:\n type: http\n url: " + }, + { + "type": "link", + "url": "https://my.fake-marquez-endpoint.com" + }, + { + "type": "text", + "text": "\n auth:\n type: some.fully.qualified.classpath" + } + ], + "border": 0 + } + ] + } + ], + "team": "T01CWUYP5AR", + "edited": { + "user": "U04AZ7992SU", + "ts": "1694466508.000000" + }, + "thread_ts": "1694466446.599329", + "reply_count": 4, + "reply_users_count": 1, + "latest_reply": "1694468262.274069", + "reply_users": [ + "U04AZ7992SU" + ], + "is_locked": false, + "subscribed": false, + "replies": [ + { + "client_msg_id": "5a0e67f9-4d57-4a3a-af1e-8a59d8d38fa8", + "type": "message", + "text": "If I log this line I can tell the TokenProvider is the class instance I would expect: ", + "user": "U04AZ7992SU", + "ts": "1694466580.365229", + "blocks": [ + { + "type": "rich_text", + "block_id": "SJEKq", + "elements": [ + { + "type": "rich_text_section", + "elements": [ + { + "type": "text", + "text": "If I log this line I can tell the TokenProvider is the class instance I would expect: " + }, + { + "type": "link", + "url": "https://github.com/OpenLineage/OpenLineage/blob/45d94fb73b5488d34b8ca544b58317382ceb3980/client/python/openlineage/client/transport/http.py#L55" + } + ] + } + ] + } + ], + "team": "T01CWUYP5AR", + "attachments": [ + { + "id": 1, + "footer_icon": "https://slack.github.com/static/img/favicon-neutral.png", + "color": "24292f", + "bot_id": "B01VA0FB340", + "app_unfurl_url": "https://github.com/OpenLineage/OpenLineage/blob/45d94fb73b5488d34b8ca544b58317382ceb3980/client/python/openlineage/client/transport/http.py#L55", + "is_app_unfurl": true, + "app_id": "A01BP7R4KNY", + "fallback": "", + "text": "```\n subclass = try_import_from_string(of_type)\n```", + "title": "", + "footer": "", + "mrkdwn_in": [ + "text" + ] + } + ], + "thread_ts": "1694466446.599329", + "parent_user_id": "U04AZ7992SU" + }, + { + "client_msg_id": "f3cbb88c-eacd-404a-884d-ffc74ba4d696", + "type": "message", + "text": "However, if I log the `token_provider` here I get the origin TokenProvider: ", + "user": "U04AZ7992SU", + "ts": "1694466674.881149", + "blocks": [ + { + "type": "rich_text", + "block_id": "2GXxm", + "elements": [ + { + "type": "rich_text_section", + "elements": [ + { + "type": "text", + "text": "However, if I log the " + }, + { + "type": "text", + "text": "token_provider", + "style": { + "code": true + } + }, + { + "type": "text", + "text": " here I get the origin TokenProvider: " + }, + { + "type": "link", + "url": "https://github.com/OpenLineage/OpenLineage/blob/45d94fb73b5488d34b8ca544b58317382ceb3980/client/python/openlineage/client/transport/http.py#L154" + } + ] + } + ] + } + ], + "team": "T01CWUYP5AR", + "attachments": [ + { + "id": 1, + "footer_icon": "https://slack.github.com/static/img/favicon-neutral.png", + "color": "24292f", + "bot_id": "B01VA0FB340", + "app_unfurl_url": "https://github.com/OpenLineage/OpenLineage/blob/45d94fb73b5488d34b8ca544b58317382ceb3980/client/python/openlineage/client/transport/http.py#L154", + "is_app_unfurl": true, + "app_id": "A01BP7R4KNY", + "fallback": "", + "text": "```\n bearer = token_provider.get_bearer()\n```", + "title": "", + "footer": "", + "mrkdwn_in": [ + "text" + ] + } + ], + "thread_ts": "1694466446.599329", + "parent_user_id": "U04AZ7992SU" + }, + { + "client_msg_id": "7a84dd3f-bd66-446b-8664-550e3e9c367b", + "type": "message", + "text": "Ah I think I see the issue. Looks like this was introduced here, we are instantiating with the base token provider here when we should be using the subclass: ", + "user": "U04AZ7992SU", + "ts": "1694467136.944659", + "blocks": [ + { + "type": "rich_text", + "block_id": "cUROS", + "elements": [ + { + "type": "rich_text_section", + "elements": [ + { + "type": "text", + "text": "Ah I think I see the issue. Looks like this was introduced here, we are instantiating with the base token provider here when we should be using the subclass: " + }, + { + "type": "link", + "url": "https://github.com/OpenLineage/OpenLineage/pull/1869/files#diff-2f8ea6f9a22b5567de8ab56c6a63da8e7adf40cb436ee5e7e6b16e70a82afe05R57" + } + ] + } + ] + } + ], + "team": "T01CWUYP5AR", + "thread_ts": "1694466446.599329", + "parent_user_id": "U04AZ7992SU" + }, + { + "type": "message", + "subtype": "thread_broadcast", + "text": "Opened a PR for this here: ", + "user": "U04AZ7992SU", + "ts": "1694468262.274069", + "thread_ts": "1694466446.599329", + "root": { + "client_msg_id": "375d0fac-c7ba-4741-bf3b-4194cf48df0e", + "type": "message", + "text": "I’m seeing some odd behavior with my http transport when upgrading airflow/openlineage-airflow from 2.3.2 -> 2.6.3 and 0.24.0 -> 0.28.0. Previously I had a config like this that let me provide my own auth tokens. However, after upgrading I’m getting a 401 from the endpoint and further debugging seems to reveal that we’re not using the token provided in my TokenProvider. Does anyone know if something changed between these versions that could be causing this? (more details in :thread: )\n```transport:\n type: http\n url: \n auth:\n type: some.fully.qualified.classpath```", + "user": "U04AZ7992SU", + "ts": "1694466446.599329", + "blocks": [ + { + "type": "rich_text", + "block_id": "zNf8V", + "elements": [ + { + "type": "rich_text_section", + "elements": [ + { + "type": "text", + "text": "I’m seeing some odd behavior with my http transport when upgrading airflow/openlineage-airflow from 2.3.2 -> 2.6.3 and 0.24.0 -> 0.28.0. Previously I had a config like this that let me provide my own auth tokens. However, after upgrading I’m getting a 401 from the endpoint and further debugging seems to reveal that we’re not using the token provided in my TokenProvider. Does anyone know if something changed between these versions that could be causing this? (more details in " + }, + { + "type": "emoji", + "name": "thread", + "unicode": "1f9f5" + }, + { + "type": "text", + "text": " )\n" + } + ] + }, + { + "type": "rich_text_preformatted", + "elements": [ + { + "type": "text", + "text": "transport:\n type: http\n url: " + }, + { + "type": "link", + "url": "https://my.fake-marquez-endpoint.com" + }, + { + "type": "text", + "text": "\n auth:\n type: some.fully.qualified.classpath" + } + ], + "border": 0 + } + ] + } + ], + "team": "T01CWUYP5AR", + "edited": { + "user": "U04AZ7992SU", + "ts": "1694466508.000000" + }, + "thread_ts": "1694466446.599329", + "reply_count": 4, + "reply_users_count": 1, + "latest_reply": "1694468262.274069", + "reply_users": [ + "U04AZ7992SU" + ], + "is_locked": false, + "subscribed": false + }, + "blocks": [ + { + "type": "rich_text", + "block_id": "Sua8X", + "elements": [ + { + "type": "rich_text_section", + "elements": [ + { + "type": "text", + "text": "Opened a PR for this here: " + }, + { + "type": "link", + "url": "https://github.com/OpenLineage/OpenLineage/pull/2100" + } + ] + } + ] + } + ], + "attachments": [ + { + "id": 1, + "footer_icon": "https://slack.github.com/static/img/favicon-neutral.png", + "ts": 1694468121, + "color": "36a64f", + "bot_id": "B01VA0FB340", + "app_unfurl_url": "https://github.com/OpenLineage/OpenLineage/pull/2100", + "is_app_unfurl": true, + "app_id": "A01BP7R4KNY", + "fallback": "#2100 python: fix custom http transport TokenProvider", + "text": "*Problem*\n\nLooks like in we introduced a bug where we would always use the base `TokenProvider` class instead of a given custom token provider, even if the validation conditions passed.\n\n*Solution*\n\nRevert this to instantiate the subclass\n\n☐ Your change modifies the OpenLineage model\n☐ Your change modifies one or more OpenLineage \n\nIf you're contributing a new integration, please specify the scope of the integration and how/where it has been tested (e.g., Apache Spark integration supports `S3` and `GCS` filesystem operations, tested with AWS EMR).\n\n*One-line summary:*\n*Checklist*\n\n☑︎ You've your work\n☑︎ Your pull request title follows our \n☐ Your changes are accompanied by tests (_if relevant_)\n☑︎ Your change contains a and is self-contained\n☐ You've updated any relevant documentation (_if relevant_)\n☐ Your comment includes a one-liner for the changelog about the specific purpose of the change (_if necessary_)\n☐ You've versioned the core OpenLineage model or facets according to (_if relevant_)\n☐ You've added a to source files (_if relevant_)\n\n* * *\n\nSPDX-License-Identifier: Apache-2.0 \nCopyright 2018-2023 contributors to the OpenLineage project", + "title": "#2100 python: fix custom http transport TokenProvider", + "title_link": "https://github.com/OpenLineage/OpenLineage/pull/2100", + "footer": "", + "fields": [ + { + "value": "client/python", + "title": "Labels", + "short": true + }, + { + "value": "1", + "title": "Comments", + "short": true + } + ], + "mrkdwn_in": [ + "text" + ] + } + ], + "client_msg_id": "880a3f24-c44a-4818-ad1c-2eef97fec037", + "reactions": [ + { + "name": "heart", + "users": [ + "U01DCLP0GU9" + ], + "count": 1 + } + ] + } + ] + }, + { + "client_msg_id": "b03bb166-9c8f-4e02-9630-39985ec4a91a", + "type": "message", + "text": "\nThe first Toronto OpenLineage Meetup, featuring a presentation by recent adopter , is just one week away. On the agenda:\n1. *Evolution of spec presentation/discussion (project background/history)*\n2. *State of the community*\n3. *Integrating OpenLineage with (by special guests & )*\n4. *Spark/Column lineage update*\n5. *Airflow Provider update*\n6. *Roadmap Discussion*\n*Find more details and RSVP *.", + "user": "U02LXF3HUN7", + "ts": "1694441261.486759", + "blocks": [ + { + "type": "rich_text", + "block_id": "t94g1", + "elements": [ + { + "type": "rich_text_section", + "elements": [ + { + "type": "broadcast", + "range": "channel" + }, + { + "type": "text", + "text": "\nThe first Toronto OpenLineage Meetup, featuring a presentation by recent adopter " + }, + { + "type": "link", + "url": "https://metaphor.io/", + "text": "Metaphor" + }, + { + "type": "text", + "text": ", is just one week away. On the agenda:\n" + } + ] + }, + { + "type": "rich_text_list", + "elements": [ + { + "type": "rich_text_section", + "elements": [ + { + "type": "text", + "text": "Evolution of spec presentation/discussion (project background/history)", + "style": { + "bold": true + } + } + ] + }, + { + "type": "rich_text_section", + "elements": [ + { + "type": "text", + "text": "State of the community", + "style": { + "bold": true + } + } + ] + }, + { + "type": "rich_text_section", + "elements": [ + { + "type": "text", + "text": "Integrating OpenLineage with ", + "style": { + "bold": true + } + }, + { + "type": "link", + "url": "https://metaphor.io/", + "text": "Metaphor", + "style": { + "bold": true + } + }, + { + "type": "text", + "text": " (by special guests ", + "style": { + "bold": true + } + }, + { + "type": "link", + "url": "https://www.linkedin.com/in/yeliu84/", + "text": "Ye", + "style": { + "bold": true + } + }, + { + "type": "text", + "text": " & ", + "style": { + "bold": true + } + }, + { + "type": "link", + "url": "https://www.linkedin.com/in/ivanperepelitca/", + "text": "Ivan", + "style": { + "bold": true + } + }, + { + "type": "text", + "text": ")", + "style": { + "bold": true + } + } + ] + }, + { + "type": "rich_text_section", + "elements": [ + { + "type": "text", + "text": "Spark/Column lineage update", + "style": { + "bold": true + } + } + ] + }, + { + "type": "rich_text_section", + "elements": [ + { + "type": "text", + "text": "Airflow Provider update", + "style": { + "bold": true + } + } + ] + }, + { + "type": "rich_text_section", + "elements": [ + { + "type": "text", + "text": "Roadmap Discussion", + "style": { + "bold": true + } + } + ] + } + ], + "style": "ordered", + "indent": 0, + "border": 0 + }, + { + "type": "rich_text_section", + "elements": [ + { + "type": "text", + "text": "Find more details and RSVP ", + "style": { + "bold": true + } + }, + { + "type": "link", + "url": "https://www.meetup.com/openlineage/events/295488014/?utm_medium=referral&utm_campaign=share-btn_savedevents_share_modal&utm_source=link", + "text": "here", + "style": { + "bold": true + } + }, + { + "type": "text", + "text": "." + } + ] + } + ] + } + ], + "team": "T01CWUYP5AR", + "attachments": [ + { + "image_url": "https://static.metaphor.io/preview.jpg", + "image_width": 719, + "image_height": 378, + "image_bytes": 122301, + "from_url": "https://metaphor.io/", + "id": 1, + "original_url": "https://metaphor.io/", + "fallback": "Metaphor - The Social Platform for Data", + "text": "Making Data Actionable, At Scale - Designed for data teams building cloud-native, self-service data platforms for their business users. Explore our Data Governance, Data Lineage, Data Discovery, and Data Trust capabilities today.", + "title": "Metaphor - The Social Platform for Data", + "title_link": "https://metaphor.io/", + "service_name": "metaphor.io" + }, + { + "from_url": "https://www.meetup.com/openlineage/events/295488014/?utm_medium=referral&utm_campaign=share-btn_savedevents_share_modal&utm_source=link", + "image_url": "https://secure.meetupstatic.com/photos/event/5/4/2/d/600_515181549.jpeg", + "image_width": 600, + "image_height": 338, + "image_bytes": 16248, + "service_icon": "https://secure.meetupstatic.com/next/images/general/m_swarm_120x120.png", + "id": 2, + "original_url": "https://www.meetup.com/openlineage/events/295488014/?utm_medium=referral&utm_campaign=share-btn_savedevents_share_modal&utm_source=link", + "fallback": "Meetup: Toronto OpenLineage Meetup at Airflow Summit, Mon, Sep 18, 2023, 5:00 PM | Meetup", + "text": "Data engineers and pipeline managers know that producing data lineage – end-to-end pipeline metadata instrumented at runtime or parsed at design time – is a heavy lift with", + "title": "Toronto OpenLineage Meetup at Airflow Summit, Mon, Sep 18, 2023, 5:00 PM | Meetup", + "title_link": "https://www.meetup.com/openlineage/events/295488014/?utm_medium=referral&utm_campaign=share-btn_savedevents_share_modal&utm_source=link", + "service_name": "Meetup" + } + ], + "reactions": [ + { + "name": "raised_hands", + "users": [ + "U01HVNU6A4C", + "U0282HEEHB8", + "U01HNKK4XAM", + "U01RA9B5GG2", + "U05KKM07PJP", + "U02MK6YNAQ5", + "U05GTE94SKY" + ], + "count": 7 + } + ] + }, + { + "client_msg_id": "f78687fc-543e-4f8f-a24a-4d116a7734e7", + "type": "message", + "text": "\nThis month’s TSC meeting is next Thursday the 14th at 10am PT. On the tentative agenda:\n• announcements\n• recent releases\n• demo: Spark integration tests in Databricks runtime\n• open discussion\n• more (TBA)\nMore info and the meeting link can be found on the . All are welcome! Also, feel free to reply or DM me with discussion topics, agenda items, etc.", + "user": "U02LXF3HUN7", + "ts": "1694113940.400549", + "blocks": [ + { + "type": "rich_text", + "block_id": "Yv9ts", + "elements": [ + { + "type": "rich_text_section", + "elements": [ + { + "type": "broadcast", + "range": "channel" + }, + { + "type": "text", + "text": "\nThis month’s TSC meeting is next Thursday the 14th at 10am PT. On the tentative agenda:\n" + } + ] + }, + { + "type": "rich_text_list", + "elements": [ + { + "type": "rich_text_section", + "elements": [ + { + "type": "text", + "text": "announcements" + } + ] + }, + { + "type": "rich_text_section", + "elements": [ + { + "type": "text", + "text": "recent releases" + } + ] + }, + { + "type": "rich_text_section", + "elements": [ + { + "type": "text", + "text": "demo: Spark integration tests in Databricks runtime" + } + ] + }, + { + "type": "rich_text_section", + "elements": [ + { + "type": "text", + "text": "open discussion" + } + ] + }, + { + "type": "rich_text_section", + "elements": [ + { + "type": "text", + "text": "more (TBA)" + } + ] + } + ], + "style": "bullet", + "indent": 0, + "border": 0 + }, + { + "type": "rich_text_section", + "elements": [ + { + "type": "text", + "text": "More info and the meeting link can be found on the " + }, + { + "type": "link", + "url": "https://openlineage.io/meetings/", + "text": "website" + }, + { + "type": "text", + "text": ". All are welcome! Also, feel free to reply or DM me with discussion topics, agenda items, etc." + } + ] + } + ] + } + ], + "team": "T01CWUYP5AR", + "edited": { + "user": "U02LXF3HUN7", + "ts": "1694178288.000000" + }, + "attachments": [ + { + "from_url": "https://openlineage.io/meetings/", + "service_icon": "https://openlineage.io/img/favicon.ico", + "id": 1, + "original_url": "https://openlineage.io/meetings/", + "fallback": "TSC Meetings | OpenLineage", + "text": "The OpenLineage Technical Steering Committee meets monthly, and is open to all.", + "title": "TSC Meetings | OpenLineage", + "title_link": "https://openlineage.io/meetings/", + "service_name": "openlineage.io" + } + ], + "reactions": [ + { + "name": "+1", + "users": [ + "U01RA9B5GG2" + ], + "count": 1 + } + ] + }, + { + "client_msg_id": "c5f1a92b-646e-40de-a2ae-09d105a3e983", + "type": "message", + "text": "Has there been any conversation on the extensibility of facets/concepts? E.g.:\n• how does one extends the list of run states to add a paused/resumed state?\n• how does one extend to add a created at field?", + "user": "U0595Q78HUG", + "ts": "1694036652.124299", + "blocks": [ + { + "type": "rich_text", + "block_id": "OU1qt", + "elements": [ + { + "type": "rich_text_section", + "elements": [ + { + "type": "text", + "text": "Has there been any conversation on the extensibility of facets/concepts? E.g.:\n" + } + ] + }, + { + "type": "rich_text_list", + "elements": [ + { + "type": "rich_text_section", + "elements": [ + { + "type": "text", + "text": "how does one extends the list of run states " + }, + { + "type": "link", + "url": "https://openlineage.io/docs/spec/run-cycle" + }, + { + "type": "text", + "text": " to add a paused/resumed state?" + } + ] + }, + { + "type": "rich_text_section", + "elements": [ + { + "type": "text", + "text": "how does one extend " + }, + { + "type": "link", + "url": "https://openlineage.io/docs/spec/facets/run-facets/nominal_time" + }, + { + "type": "text", + "text": " to add a created at field?" + } + ] + } + ], + "style": "bullet", + "indent": 0, + "border": 0 + } + ] + } + ], + "team": "T01CWUYP5AR", + "attachments": [ + { + "from_url": "https://openlineage.io/docs/spec/run-cycle", + "service_icon": "https://openlineage.io/img/favicon.ico", + "id": 1, + "original_url": "https://openlineage.io/docs/spec/run-cycle", + "fallback": "The Run Cycle | OpenLineage", + "text": "The OpenLineage object model is event-based and updates provide an OpenLineage backend with details about the activities of a Job.", + "title": "The Run Cycle | OpenLineage", + "title_link": "https://openlineage.io/docs/spec/run-cycle", + "service_name": "openlineage.io" + }, + { + "from_url": "https://openlineage.io/docs/spec/facets/run-facets/nominal_time", + "service_icon": "https://openlineage.io/img/favicon.ico", + "id": 2, + "original_url": "https://openlineage.io/docs/spec/facets/run-facets/nominal_time", + "fallback": "Nominal Time Facet | OpenLineage", + "text": "The facet to describe the nominal start and end time of the run. The nominal usually means the time the job run was expected to run (like a scheduled time), and the actual time can be different.", + "title": "Nominal Time Facet | OpenLineage", + "title_link": "https://openlineage.io/docs/spec/facets/run-facets/nominal_time", + "service_name": "openlineage.io" + } + ], + "thread_ts": "1694036652.124299", + "reply_count": 3, + "reply_users_count": 2, + "latest_reply": "1694039743.460029", + "reply_users": [ + "U01DCLP0GU9", + "U0595Q78HUG" + ], + "is_locked": false, + "subscribed": false, + "replies": [ + { + "client_msg_id": "522ea318-8211-40ff-9e3e-ade1bfd05c31", + "type": "message", + "text": "Hello Bernat,\n\nThe primary mechanism to extend the model is through facets. You can either:\n• create new standard facets in the spec: \n• create custom facets defined somewhere else with a prefix in their name: \n• Update existing facets with a backward compatible change (example: adding an optional field). \nThe core spec can also be modified. Here is an example of adding \nThat being said I think more granular states like pause/resume are probably better suited in a run facet. There was an issue opened for that particular one a while ago: maybe that particular discussion can continue there.\n\nFor the nominal time facet, You could open an issue describing the use case and on community agreement follow up with a PR on the facet itself: \n(adding an optional field is backwards compatible)", + "user": "U01DCLP0GU9", + "ts": "1694039297.283689", + "blocks": [ + { + "type": "rich_text", + "block_id": "SE9Oj", + "elements": [ + { + "type": "rich_text_section", + "elements": [ + { + "type": "text", + "text": "Hello Bernat,\n\nThe primary mechanism to extend the model is through facets. You can either:\n" + } + ] + }, + { + "type": "rich_text_list", + "elements": [ + { + "type": "rich_text_section", + "elements": [ + { + "type": "text", + "text": "create new standard facets in the spec: " + }, + { + "type": "link", + "url": "https://github.com/OpenLineage/OpenLineage/tree/main/spec/facets" + } + ] + }, + { + "type": "rich_text_section", + "elements": [ + { + "type": "text", + "text": "create custom facets defined somewhere else with a prefix in their name: " + }, + { + "type": "link", + "url": "https://github.com/OpenLineage/OpenLineage/blob/main/spec/OpenLineage.md#custom-facet-naming" + } + ] + }, + { + "type": "rich_text_section", + "elements": [ + { + "type": "text", + "text": "Update existing facets with a backward compatible change (example: adding an optional field). " + } + ] + } + ], + "style": "bullet", + "indent": 0, + "border": 0 + }, + { + "type": "rich_text_section", + "elements": [ + { + "type": "text", + "text": "\nThe core spec can also be modified. Here is an example of adding " + }, + { + "type": "link", + "url": "https://github.com/OpenLineage/OpenLineage/commit/7243d916f5400dbbceaece5cda89da961ad005d3", + "text": "a state" + }, + { + "type": "text", + "text": "\nThat being said I think more granular states like pause/resume are probably better suited in a run facet. There was an issue opened for that particular one a while ago: " + }, + { + "type": "link", + "url": "https://github.com/OpenLineage/OpenLineage/issues/9" + }, + { + "type": "text", + "text": " maybe that particular discussion can continue there.\n\nFor the nominal time facet, You could open an issue describing the use case and on community agreement follow up with a PR on the facet itself: " + }, + { + "type": "link", + "url": "https://github.com/OpenLineage/OpenLineage/blob/main/spec/facets/NominalTimeRunFacet.json" + }, + { + "type": "text", + "text": "\n(adding an optional field is backwards compatible)" + } + ] + } + ] + } + ], + "team": "T01CWUYP5AR", + "thread_ts": "1694036652.124299", + "parent_user_id": "U0595Q78HUG", + "reactions": [ + { + "name": "eyes", + "users": [ + "U05HFGKEYVB" + ], + "count": 1 + } + ] + }, + { + "client_msg_id": "64582b1d-c6d5-4b4d-b3f4-169c315315b9", + "type": "message", + "text": "I see, so in general one is best copying a standard facet and maintain it under a different name. That way can be made mandatory :slightly_smiling_face: and one does not need to be blocked for a long time until there's a community agreement :thinking_face:", + "user": "U0595Q78HUG", + "ts": "1694039472.212619", + "blocks": [ + { + "type": "rich_text", + "block_id": "mJcfA", + "elements": [ + { + "type": "rich_text_section", + "elements": [ + { + "type": "text", + "text": "I see, so in general one is best copying a standard facet and maintain it under a different name. That way can be made mandatory " + }, + { + "type": "emoji", + "name": "slightly_smiling_face", + "unicode": "1f642" + }, + { + "type": "text", + "text": " and one does not need to be blocked for a long time until there's a community agreement " + }, + { + "type": "emoji", + "name": "thinking_face", + "unicode": "1f914" + } + ] + } + ] + } + ], + "team": "T01CWUYP5AR", + "thread_ts": "1694036652.124299", + "parent_user_id": "U0595Q78HUG" + }, + { + "client_msg_id": "ceeabfce-4337-4d8f-be9b-a43ab63e2a4d", + "type": "message", + "text": "Yes, The goal of custom facets is to allow you to experiment and extend the spec however you want without having to wait for approval.\nIf the custom facet is very specific to a third party project/product then it makes sense for it to stay a custom facet.\nIf it is more generic then it makes sense to add it to the core facets as part of the spec.\nHopefully community agreement can be achieved relatively quickly. Unless someone is strongly against something, it can be added without too much red tape. Typically with support in at least one of the integrations to validate the model.", + "user": "U01DCLP0GU9", + "ts": "1694039743.460029", + "blocks": [ + { + "type": "rich_text", + "block_id": "FA7ar", + "elements": [ + { + "type": "rich_text_section", + "elements": [ + { + "type": "text", + "text": "Yes, The goal of custom facets is to allow you to experiment and extend the spec however you want without having to wait for approval.\nIf the custom facet is very specific to a third party project/product then it makes sense for it to stay a custom facet.\nIf it is more generic then it makes sense to add it to the core facets as part of the spec.\nHopefully community agreement can be achieved relatively quickly. Unless someone is strongly against something, it can be added without too much red tape. Typically with support in at least one of the integrations to validate the model." + } + ] + } + ] + } + ], + "team": "T01CWUYP5AR", + "thread_ts": "1694036652.124299", + "parent_user_id": "U0595Q78HUG" + } + ] + }, + { + "client_msg_id": "78d7f8ab-38ea-4413-a07b-03c683d63028", + "type": "message", + "text": "Hello Everyone,\n\nI've been diving into the Marquez codebase and found a performance bottleneck in `JobDao.java` for the query related to `namespaceName=MyNameSpace` `limit=10` and 12s with `limit=25`. I managed to optimize it using CTEs, and the execution times dropped dramatically to 300ms (for `limit=100`) and under 100ms (for `limit=25` ) on the same cluster.\nIssue link : \n\nI believe there's even more room for optimization, especially if we adjust the `job_facets_view` to include the `namespace_name` column.\n\nWould the team be open to a PR where I share the optimized query and discuss potential further refinements? I believe these changes could significantly enhance the Marquez web UI experience.\n\nPR link : \n\nLooking forward to your feedback.", + "user": "U05HBLE7YPL", + "ts": "1694032987.624809", + "blocks": [ + { + "type": "rich_text", + "block_id": "htkZ4", + "elements": [ + { + "type": "rich_text_section", + "elements": [ + { + "type": "text", + "text": "Hello Everyone,\n\nI've been diving into the Marquez codebase and found a performance bottleneck in " + }, + { + "type": "text", + "text": "JobDao.java", + "style": { + "code": true + } + }, + { + "type": "text", + "text": " for the query related to " + }, + { + "type": "text", + "text": "namespaceName=MyNameSpace", + "style": { + "code": true + } + }, + { + "type": "text", + "text": " " + }, + { + "type": "text", + "text": "limit=10", + "style": { + "code": true + } + }, + { + "type": "text", + "text": " and 12s with " + }, + { + "type": "text", + "text": "limit=25", + "style": { + "code": true + } + }, + { + "type": "text", + "text": ". I managed to optimize it using CTEs, and the execution times dropped dramatically to 300ms (for " + }, + { + "type": "text", + "text": "limit=100", + "style": { + "code": true + } + }, + { + "type": "text", + "text": ") and under 100ms (for " + }, + { + "type": "text", + "text": "limit=25", + "style": { + "code": true + } + }, + { + "type": "text", + "text": " ) on the same cluster.\nIssue link : " + }, + { + "type": "link", + "url": "https://github.com/MarquezProject/marquez/issues/2608" + }, + { + "type": "text", + "text": "\n\nI believe there's even more room for optimization, especially if we adjust the `job_facets_view` to include the " + }, + { + "type": "text", + "text": "namespace_name", + "style": { + "code": true + } + }, + { + "type": "text", + "text": " column.\n\nWould the team be open to a PR where I share the optimized query and discuss potential further refinements? I believe these changes could significantly enhance the Marquez web UI experience.\n\nPR link : " + }, + { + "type": "link", + "url": "https://github.com/MarquezProject/marquez/pull/2609" + }, + { + "type": "text", + "text": "\n\nLooking forward to your feedback." + } + ] + } + ] + } + ], + "team": "T01CWUYP5AR", + "edited": { + "user": "U05HBLE7YPL", + "ts": "1694033114.000000" + }, + "attachments": [ + { + "id": 1, + "footer_icon": "https://slack.github.com/static/img/favicon-neutral.png", + "ts": 1694025187, + "color": "36a64f", + "bot_id": "B01VA0FB340", + "app_unfurl_url": "https://github.com/MarquezProject/marquez/issues/2608", + "is_app_unfurl": true, + "app_id": "A01BP7R4KNY", + "fallback": "#2608 Performance Issue with Query on Large Data Sets in JobDao.java", + "title": "#2608 Performance Issue with Query on Large Data Sets in JobDao.java", + "title_link": "https://github.com/MarquezProject/marquez/issues/2608", + "footer": "", + "mrkdwn_in": [ + "text" + ] + }, + { + "id": 2, + "footer_icon": "https://slack.github.com/static/img/favicon-neutral.png", + "ts": 1694032737, + "color": "36a64f", + "bot_id": "B01VA0FB340", + "app_unfurl_url": "https://github.com/MarquezProject/marquez/pull/2609", + "is_app_unfurl": true, + "app_id": "A01BP7R4KNY", + "fallback": "#2609 Perf/improve jobdao query", + "title": "#2609 Perf/improve jobdao query", + "title_link": "https://github.com/MarquezProject/marquez/pull/2609", + "footer": "", + "mrkdwn_in": [ + "text" + ] + } + ], + "thread_ts": "1694032987.624809", + "reply_count": 1, + "reply_users_count": 1, + "latest_reply": "1694037781.710149", + "reply_users": [ + "U02S6F54MAB" + ], + "is_locked": false, + "subscribed": false, + "reactions": [ + { + "name": "fire", + "users": [ + "U02S6F54MAB", + "U01HNKK4XAM", + "U02MK6YNAQ5", + "U01RA9B5GG2" + ], + "count": 4 + } + ], + "replies": [ + { + "client_msg_id": "e841183d-05e0-4ab1-898a-20856329f92a", + "type": "message", + "text": "<@U01DCMDFHBK> wdyt?", + "user": "U02S6F54MAB", + "ts": "1694037781.710149", + "blocks": [ + { + "type": "rich_text", + "block_id": "o6dVs", + "elements": [ + { + "type": "rich_text_section", + "elements": [ + { + "type": "user", + "user_id": "U01DCMDFHBK" + }, + { + "type": "text", + "text": " wdyt?" + } + ] + } + ] + } + ], + "team": "T01CWUYP5AR", + "thread_ts": "1694032987.624809", + "parent_user_id": "U05HBLE7YPL" + } + ] + }, + { + "client_msg_id": "3e9318e1-fd7b-44e0-bf1f-8d5e0cb45d8f", + "type": "message", + "text": "it looks like my dynamic task mapping in Airflow has the same run ID in marquez, so even if I am processing 100 files, there is only one version of the data. is there a way to have a separate version of each dynamic task so I can track the filename etc?", + "user": "U05NMJ0NBUK", + "ts": "1693877705.781699", + "blocks": [ + { + "type": "rich_text", + "block_id": "1Ox3", + "elements": [ + { + "type": "rich_text_section", + "elements": [ + { + "type": "text", + "text": "it looks like my dynamic task mapping in Airflow has the same run ID in marquez, so even if I am processing 100 files, there is only one version of the data. is there a way to have a separate version of each dynamic task so I can track the filename etc?" + } + ] + } + ] + } + ], + "team": "T01CWUYP5AR", + "thread_ts": "1693877705.781699", + "reply_count": 17, + "reply_users_count": 2, + "latest_reply": "1694393357.711739", + "reply_users": [ + "U02S6F54MAB", + "U05NMJ0NBUK" + ], + "is_locked": false, + "subscribed": false, + "replies": [ + { + "client_msg_id": "e3e7a32e-a0c7-4253-9a1b-9fb29b7a1642", + "type": "message", + "text": "`map_index` should be indeed included when calculating run ID (it’s deterministic in Airflow integration)\nwhat version of Airflow are you using btw?", + "user": "U02S6F54MAB", + "ts": "1693918497.791789", + "blocks": [ + { + "type": "rich_text", + "block_id": "eoO", + "elements": [ + { + "type": "rich_text_section", + "elements": [ + { + "type": "text", + "text": "map_index", + "style": { + "code": true + } + }, + { + "type": "text", + "text": " should be indeed included when calculating run ID (it’s deterministic in Airflow integration)\nwhat version of Airflow are you using btw?" + } + ] + } + ] + } + ], + "team": "T01CWUYP5AR", + "thread_ts": "1693877705.781699", + "parent_user_id": "U05NMJ0NBUK" + }, + { + "client_msg_id": "744dc85b-d6b8-4746-8f91-67321a39994a", + "type": "message", + "text": "2.7.0\n\nI do see this error log in all of my dynamic tasks which might explain it:\n\n```[2023-09-05, 00:31:57 UTC] {manager.py:200} ERROR - Extractor returns non-valid metadata: None\n[2023-09-05, 00:31:57 UTC] {utils.py:401} ERROR - cannot import name 'get_operator_class' from 'airflow.providers.openlineage.utils' (/home/airflow/.local/lib/python3.11/site-packages/airflow/providers/openlineage/utils/__init__.py)\nTraceback (most recent call last):\n File \"/home/airflow/.local/lib/python3.11/site-packages/airflow/providers/openlineage/utils/utils.py\", line 399, in wrapper\n return f(*args, **kwargs)\n ^^^^^^^^^^^^^^^^^^\n File \"/home/airflow/.local/lib/python3.11/site-packages/airflow/providers/openlineage/plugins/listener.py\", line 93, in on_running\n **get_custom_facets(task_instance),\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n File \"/home/airflow/.local/lib/python3.11/site-packages/airflow/providers/openlineage/utils/utils.py\", line 148, in get_custom_facets\n custom_facets[\"airflow_mappedTask\"] = AirflowMappedTaskRunFacet.from_task_instance(task_instance)\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n File \"/home/airflow/.local/lib/python3.11/site-packages/airflow/providers/openlineage/plugins/facets.py\", line 36, in from_task_instance\n from airflow.providers.openlineage.utils import get_operator_class\nImportError: cannot import name 'get_operator_class' from 'airflow.providers.openlineage.utils' (/home/airflow/.local/lib/python3.11/site-packages/airflow/providers/openlineage/utils/__init__.py)```", + "user": "U05NMJ0NBUK", + "ts": "1693919054.875009", + "blocks": [ + { + "type": "rich_text", + "block_id": "rlf/", + "elements": [ + { + "type": "rich_text_section", + "elements": [ + { + "type": "text", + "text": "2.7.0\n\nI do see this error log in all of my dynamic tasks which might explain it:\n\n" + } + ] + }, + { + "type": "rich_text_preformatted", + "elements": [ + { + "type": "text", + "text": "[2023-09-05, 00:31:57 UTC] {manager.py:200} ERROR - Extractor returns non-valid metadata: None\n[2023-09-05, 00:31:57 UTC] {utils.py:401} ERROR - cannot import name 'get_operator_class' from 'airflow.providers.openlineage.utils' (/home/airflow/.local/lib/python3.11/site-packages/airflow/providers/openlineage/utils/__init__.py)\nTraceback (most recent call last):\n File \"/home/airflow/.local/lib/python3.11/site-packages/airflow/providers/openlineage/utils/utils.py\", line 399, in wrapper\n return f(*args, **kwargs)\n ^^^^^^^^^^^^^^^^^^\n File \"/home/airflow/.local/lib/python3.11/site-packages/airflow/providers/openlineage/plugins/listener.py\", line 93, in on_running\n **get_custom_facets(task_instance),\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n File \"/home/airflow/.local/lib/python3.11/site-packages/airflow/providers/openlineage/utils/utils.py\", line 148, in get_custom_facets\n custom_facets[\"airflow_mappedTask\"] = AirflowMappedTaskRunFacet.from_task_instance(task_instance)\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n File \"/home/airflow/.local/lib/python3.11/site-packages/airflow/providers/openlineage/plugins/facets.py\", line 36, in from_task_instance\n from airflow.providers.openlineage.utils import get_operator_class\nImportError: cannot import name 'get_operator_class' from 'airflow.providers.openlineage.utils' (/home/airflow/.local/lib/python3.11/site-packages/airflow/providers/openlineage/utils/__init__.py)" + } + ], + "border": 0 + } + ] + } + ], + "team": "T01CWUYP5AR", + "thread_ts": "1693877705.781699", + "parent_user_id": "U05NMJ0NBUK" + }, + { + "client_msg_id": "8606bfed-db0d-45ad-b6ac-1a399568d656", + "type": "message", + "text": "I only have a few custom operators with the on_complete facet so I think this is a built in one - it runs before my task custom logs for example", + "user": "U05NMJ0NBUK", + "ts": "1693919134.573409", + "blocks": [ + { + "type": "rich_text", + "block_id": "NJh", + "elements": [ + { + "type": "rich_text_section", + "elements": [ + { + "type": "text", + "text": "I only have a few custom operators with the on_complete facet so I think this is a built in one - it runs before my task custom logs for example" + } + ] + } + ] + } + ], + "team": "T01CWUYP5AR", + "thread_ts": "1693877705.781699", + "parent_user_id": "U05NMJ0NBUK" + }, + { + "client_msg_id": "5a721b5e-ac98-45ce-b0b8-79bc03906086", + "type": "message", + "text": "and any time I messed up my custom facet, the error would be at the bottom of the logs. this is on top, probably an on_start facet?", + "user": "U05NMJ0NBUK", + "ts": "1693919165.643139", + "blocks": [ + { + "type": "rich_text", + "block_id": "xZGQ", + "elements": [ + { + "type": "rich_text_section", + "elements": [ + { + "type": "text", + "text": "and any time I messed up my custom facet, the error would be at the bottom of the logs. this is on top, probably an on_start facet?" + } + ] + } + ] + } + ], + "team": "T01CWUYP5AR", + "thread_ts": "1693877705.781699", + "parent_user_id": "U05NMJ0NBUK" + }, + { + "client_msg_id": "6b30cc6d-d096-4c95-be57-856255f2bc11", + "type": "message", + "text": "seems like some circular import", + "user": "U02S6F54MAB", + "ts": "1693919792.748089", + "blocks": [ + { + "type": "rich_text", + "block_id": "PpJ6", + "elements": [ + { + "type": "rich_text_section", + "elements": [ + { + "type": "text", + "text": "seems like some circular import" + } + ] + } + ] + } + ], + "team": "T01CWUYP5AR", + "thread_ts": "1693877705.781699", + "parent_user_id": "U05NMJ0NBUK" + }, + { + "client_msg_id": "81770407-4d16-4f26-a14f-6113f6328317", + "type": "message", + "text": "I just tested it manually, it’s a bug in OL provider. let me fix that", + "user": "U02S6F54MAB", + "ts": "1693919987.802519", + "blocks": [ + { + "type": "rich_text", + "block_id": "JZE", + "elements": [ + { + "type": "rich_text_section", + "elements": [ + { + "type": "text", + "text": "I just tested it manually, it’s a bug in OL provider. let me fix that" + } + ] + } + ] + } + ], + "team": "T01CWUYP5AR", + "thread_ts": "1693877705.781699", + "parent_user_id": "U05NMJ0NBUK" + }, + { + "client_msg_id": "e5c44698-d548-4e9b-bc54-bfed500771f4", + "type": "message", + "text": "cool, thanks. I am glad it is just a bug, I was afraid dynamic tasks were not supported for a minute there", + "user": "U05NMJ0NBUK", + "ts": "1693925608.644659", + "blocks": [ + { + "type": "rich_text", + "block_id": "AbK/7", + "elements": [ + { + "type": "rich_text_section", + "elements": [ + { + "type": "text", + "text": "cool, thanks. I am glad it is just a bug, I was afraid dynamic tasks were not supported for a minute there" + } + ] + } + ] + } + ], + "team": "T01CWUYP5AR", + "thread_ts": "1693877705.781699", + "parent_user_id": "U05NMJ0NBUK" + }, + { + "client_msg_id": "778e62b6-2296-4d74-a8a3-110dc66f8725", + "type": "message", + "text": "how do the provider updates work? they can be released in between Airflow releases and issues for them are raised on the main Airflow repo?", + "user": "U05NMJ0NBUK", + "ts": "1694101580.250669", + "blocks": [ + { + "type": "rich_text", + "block_id": "Yh7yu", + "elements": [ + { + "type": "rich_text_section", + "elements": [ + { + "type": "text", + "text": "how do the provider updates work? they can be released in between Airflow releases and issues for them are raised on the main Airflow repo?" + } + ] + } + ] + } + ], + "team": "T01CWUYP5AR", + "thread_ts": "1693877705.781699", + "parent_user_id": "U05NMJ0NBUK" + }, + { + "client_msg_id": "4da90885-da4c-4e50-aab2-b2c4b524c000", + "type": "message", + "text": "generally speaking anything related to OL-Airflow should be placed to Airflow repo, important changes/bug fixes would be implemented in OL repo as well", + "user": "U02S6F54MAB", + "ts": "1694101807.839469", + "blocks": [ + { + "type": "rich_text", + "block_id": "BC2mn", + "elements": [ + { + "type": "rich_text_section", + "elements": [ + { + "type": "text", + "text": "generally speaking anything related to OL-Airflow should be placed to Airflow repo, important changes/bug fixes would be implemented in OL repo as well" + } + ] + } + ] + } + ], + "team": "T01CWUYP5AR", + "thread_ts": "1693877705.781699", + "parent_user_id": "U05NMJ0NBUK" + }, + { + "client_msg_id": "e9894bbb-1fd0-4415-b0da-8032c4d9de4f", + "type": "message", + "text": "got it, thanks", + "user": "U05NMJ0NBUK", + "ts": "1694115631.653209", + "blocks": [ + { + "type": "rich_text", + "block_id": "F4swH", + "elements": [ + { + "type": "rich_text_section", + "elements": [ + { + "type": "text", + "text": "got it, thanks" + } + ] + } + ] + } + ], + "team": "T01CWUYP5AR", + "thread_ts": "1693877705.781699", + "parent_user_id": "U05NMJ0NBUK" + }, + { + "client_msg_id": "0e789d74-721b-4a4e-bd78-923bdb6c1a35", + "type": "message", + "text": "is there a way for me to install the openlineage provider based on the commit you made to fix the circular imports?\n\ni was going to try to install from Airflow main branch but didnt want to mess anything up", + "user": "U05NMJ0NBUK", + "ts": "1694130226.955559", + "blocks": [ + { + "type": "rich_text", + "block_id": "B0iPX", + "elements": [ + { + "type": "rich_text_section", + "elements": [ + { + "type": "text", + "text": "is there a way for me to install the openlineage provider based on the commit you made to fix the circular imports?\n\ni was going to try to install from Airflow main branch but didnt want to mess anything up" + } + ] + } + ] + } + ], + "team": "T01CWUYP5AR", + "thread_ts": "1693877705.781699", + "parent_user_id": "U05NMJ0NBUK" + }, + { + "client_msg_id": "db1fce8f-bc50-4191-81d0-8600b7f0047c", + "type": "message", + "text": "I saw it was merged to airflow main but it is not in 2.7.1 and there is no 1.0.3 provider version yet, so I wondered if I could manually install it for the time being", + "user": "U05NMJ0NBUK", + "ts": "1694130279.428999", + "blocks": [ + { + "type": "rich_text", + "block_id": "DkrRa", + "elements": [ + { + "type": "rich_text_section", + "elements": [ + { + "type": "text", + "text": "I saw it was merged to airflow main but it is not in 2.7.1 and there is no 1.0.3 provider version yet, so I wondered if I could manually install it for the time being" + } + ] + } + ] + } + ], + "team": "T01CWUYP5AR", + "thread_ts": "1693877705.781699", + "parent_user_id": "U05NMJ0NBUK" + }, + { + "client_msg_id": "15b65d6f-0c6f-4664-8ae7-849d70576c4b", + "type": "message", + "text": "\nbuilding the provider package on your own could be best idea probably? that depends on how you manage your Airflow instance", + "user": "U02S6F54MAB", + "ts": "1694166348.933729", + "blocks": [ + { + "type": "rich_text", + "block_id": "LBF0r", + "elements": [ + { + "type": "rich_text_section", + "elements": [ + { + "type": "link", + "url": "https://github.com/apache/airflow/blob/main/BREEZE.rst#preparing-provider-packages" + }, + { + "type": "text", + "text": "\nbuilding the provider package on your own could be best idea probably? that depends on how you manage your Airflow instance" + } + ] + } + ] + } + ], + "team": "T01CWUYP5AR", + "thread_ts": "1693877705.781699", + "parent_user_id": "U05NMJ0NBUK" + }, + { + "client_msg_id": "cf630c89-866d-4825-9b44-233ee9dba474", + "type": "message", + "text": "there's 1.1.0rc1 btw", + "user": "U02S6F54MAB", + "ts": "1694188913.801429", + "blocks": [ + { + "type": "rich_text", + "block_id": "1QZeR", + "elements": [ + { + "type": "rich_text_section", + "elements": [ + { + "type": "text", + "text": "there's 1.1.0rc1 btw" + } + ] + } + ] + } + ], + "team": "T01CWUYP5AR", + "thread_ts": "1693877705.781699", + "parent_user_id": "U05NMJ0NBUK" + }, + { + "client_msg_id": "f2941a89-0ff4-475d-a429-05c9d1ee2958", + "type": "message", + "text": "perfect, thanks. I got started with breeze but then stopped haha", + "user": "U05NMJ0NBUK", + "ts": "1694195084.901899", + "blocks": [ + { + "type": "rich_text", + "block_id": "BgOk/", + "elements": [ + { + "type": "rich_text_section", + "elements": [ + { + "type": "text", + "text": "perfect, thanks. I got started with breeze but then stopped haha" + } + ] + } + ] + } + ], + "team": "T01CWUYP5AR", + "thread_ts": "1693877705.781699", + "parent_user_id": "U05NMJ0NBUK", + "reactions": [ + { + "name": "+1", + "users": [ + "U02S6F54MAB" + ], + "count": 1 + } + ] + }, + { + "client_msg_id": "76841536-72fe-4547-aeb2-01e32b929bcd", + "type": "message", + "text": "The dynamic task mapping error is gone, I did run into this:\n\nFile \"/home/airflow/.local/lib/python3.11/site-packages/airflow/providers/openlineage/extractors/base.py\", line 70, in disabled_operators\n operator.strip() for operator in conf.get(\"openlineage\", \"disabled_for_operators\").split(\";\")\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n File \"/home/airflow/.local/lib/python3.11/site-packages/airflow/configuration.py\", line 1065, in get\n raise AirflowConfigException(f\"section/key [{section}/{key}] not found in config\")\n\nI am redeploying now with that option added to my config. I guess it did not use the default which should be \"\"", + "user": "U05NMJ0NBUK", + "ts": "1694392140.545519", + "blocks": [ + { + "type": "rich_text", + "block_id": "3TrVA", + "elements": [ + { + "type": "rich_text_section", + "elements": [ + { + "type": "text", + "text": "The dynamic task mapping error is gone, I did run into this:\n\nFile \"/home/airflow/.local/lib/python3.11/site-packages/airflow/providers/openlineage/extractors/base.py\", line 70, in disabled_operators\n operator.strip() for operator in conf.get(\"openlineage\", \"disabled_for_operators\").split(\";\")\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n File \"/home/airflow/.local/lib/python3.11/site-packages/airflow/configuration.py\", line 1065, in get\n raise AirflowConfigException(f\"section/key [{section}/{key}] not found in config\")\n\nI am redeploying now with that option added to my config. I guess it did not use the default which should be \"\"" + } + ] + } + ] + } + ], + "team": "T01CWUYP5AR", + "thread_ts": "1693877705.781699", + "parent_user_id": "U05NMJ0NBUK" + }, + { + "client_msg_id": "71d6750a-2e77-4315-be0f-3faff3a6ebaa", + "type": "message", + "text": "added \"disabled_for_operators\" to my openlineage config and it worked (using Airflow helm chart - not sure if that means there is an error because the value I provided should just be the default value, not sure why I needed to explicitly specify it)\n\n openlineage:\n disabled_for_operators: \"\"\n ...\n\n\nthis is so much better and makes a lot more sense. most of my tasks are dynamic so I was missing a lot of metadata before the fix, thanks!", + "user": "U05NMJ0NBUK", + "ts": "1694393357.711739", + "blocks": [ + { + "type": "rich_text", + "block_id": "2+P6O", + "elements": [ + { + "type": "rich_text_section", + "elements": [ + { + "type": "text", + "text": "added \"disabled_for_operators\" to my openlineage config and it worked (using Airflow helm chart - not sure if that means there is an error because the value I provided should just be the default value, not sure why I needed to explicitly specify it)\n\n openlineage:\n disabled_for_operators: \"\"\n ...\n\n\nthis is so much better and makes a lot more sense. most of my tasks are dynamic so I was missing a lot of metadata before the fix, thanks!" + } + ] + } + ] + } + ], + "team": "T01CWUYP5AR", + "thread_ts": "1693877705.781699", + "parent_user_id": "U05NMJ0NBUK" + } + ] + }, + { + "client_msg_id": "f45db95f-7a2c-4f68-8cd9-e0d54d599cfb", + "type": "message", + "text": "Also, another small clarification is that when using `MergeIntoCommand`, I'm receiving the lineage events on the backend, but I cannot seem to find any logging of the payload when I enable debug mode in openlineage. I remember there was a similar issue reported by another user in the past. May I check if it might be possible to help with this? It's making debugging quite hard for these cases. Thanks!", + "user": "U04EZ2LPDV4", + "ts": "1693823945.734419", + "blocks": [ + { + "type": "rich_text", + "block_id": "7hi", + "elements": [ + { + "type": "rich_text_section", + "elements": [ + { + "type": "text", + "text": "Also, another small clarification is that when using " + }, + { + "type": "text", + "text": "MergeIntoCommand", + "style": { + "code": true + } + }, + { + "type": "text", + "text": ", I'm receiving the lineage events on the backend, but I cannot seem to find any logging of the payload when I enable debug mode in openlineage. I remember there was a similar issue reported by another user in the past. May I check if it might be possible to help with this? It's making debugging quite hard for these cases. Thanks!" + } + ] + } + ] + } + ], + "team": "T01CWUYP5AR", + "thread_ts": "1693823945.734419", + "reply_count": 8, + "reply_users_count": 3, + "latest_reply": "1693907271.993249", + "reply_users": [ + "U01RA9B5GG2", + "U04EZ2LPDV4", + "U02MK6YNAQ5" + ], + "is_locked": false, + "subscribed": false, + "replies": [ + { + "client_msg_id": "756996c8-a47e-4a4f-8b84-3915be1e9467", + "type": "message", + "text": "I think it only depends on log4j configuration", + "user": "U01RA9B5GG2", + "ts": "1693824852.378539", + "blocks": [ + { + "type": "rich_text", + "block_id": "eP1pE", + "elements": [ + { + "type": "rich_text_section", + "elements": [ + { + "type": "text", + "text": "I think it only depends on log4j configuration" + } + ] + } + ] + } + ], + "team": "T01CWUYP5AR", + "edited": { + "user": "U01RA9B5GG2", + "ts": "1693824855.000000" + }, + "thread_ts": "1693823945.734419", + "parent_user_id": "U04EZ2LPDV4" + }, + { + "client_msg_id": "0eda9409-a5c3-4e10-b654-55bd5257dc70", + "type": "message", + "text": "```# Set everything to be logged to the console\nlog4j.rootCategory=INFO, console\nlog4j.appender.console=org.apache.log4j.ConsoleAppender\nlog4j.appender.console.target=System.err\nlog4j.appender.console.layout=org.apache.log4j.PatternLayout\nlog4j.appender.console.layout.ConversionPattern=%d{yy/MM/dd HH:mm:ss} %p %c{1}: %m%n\n\n# set the log level for the openlineage spark library\nlog4j.logger.io.openlineage.spark=DEBUG```\nthis is what we have in `log4j.properties` in test environment and it works", + "user": "U01RA9B5GG2", + "ts": "1693825035.249249", + "blocks": [ + { + "type": "rich_text", + "block_id": "Er3", + "elements": [ + { + "type": "rich_text_preformatted", + "elements": [ + { + "type": "text", + "text": "# Set everything to be logged to the console\nlog4j.rootCategory=INFO, console\nlog4j.appender.console=org.apache.log4j.ConsoleAppender\nlog4j.appender.console.target=System.err\nlog4j.appender.console.layout=org.apache.log4j.PatternLayout\nlog4j.appender.console.layout.ConversionPattern=%d{yy/MM/dd HH:mm:ss} %p %c{1}: %m%n\n\n# set the log level for the openlineage spark library\nlog4j.logger.io.openlineage.spark=DEBUG" + } + ], + "border": 0 + }, + { + "type": "rich_text_section", + "elements": [ + { + "type": "text", + "text": "this is what we have in " + }, + { + "type": "text", + "text": "log4j.properties", + "style": { + "code": true + } + }, + { + "type": "text", + "text": " in test environment and it works" + } + ] + } + ] + } + ], + "team": "T01CWUYP5AR", + "thread_ts": "1693823945.734419", + "parent_user_id": "U04EZ2LPDV4" + }, + { + "client_msg_id": "6682df1c-6ce5-47f2-bde2-547fe609b22d", + "type": "message", + "text": "Hmm... I can see the logs for the other commands, like createViewCommand etc. I just cannot see it for any of the delta runs", + "user": "U04EZ2LPDV4", + "ts": "1693841291.922409", + "blocks": [ + { + "type": "rich_text", + "block_id": "fPkW2", + "elements": [ + { + "type": "rich_text_section", + "elements": [ + { + "type": "text", + "text": "Hmm... I can see the logs for the other commands, like createViewCommand etc. I just cannot see it for any of the delta runs" + } + ] + } + ] + } + ], + "team": "T01CWUYP5AR", + "thread_ts": "1693823945.734419", + "parent_user_id": "U04EZ2LPDV4" + }, + { + "client_msg_id": "322c8ef9-6f20-4555-9a85-ae592f74d230", + "type": "message", + "text": "that's interesting. So, logging is done here: and this code is unaware of delta.\n\nThe possible problem could be filtering delta events (which we do bcz of delta being noisy)", + "user": "U02MK6YNAQ5", + "ts": "1693899183.161579", + "blocks": [ + { + "type": "rich_text", + "block_id": "iMhU", + "elements": [ + { + "type": "rich_text_section", + "elements": [ + { + "type": "text", + "text": "that's interesting. So, logging is done here: " + }, + { + "type": "link", + "url": "https://github.com/OpenLineage/OpenLineage/blob/main/integration/spark/app/src/main/java/io/openlineage/spark/agent/EventEmitter.java#L63" + }, + { + "type": "text", + "text": " and this code is unaware of delta.\n\nThe possible problem could be filtering delta events (which we do bcz of delta being noisy)" + } + ] + } + ] + } + ], + "team": "T01CWUYP5AR", + "attachments": [ + { + "id": 1, + "footer_icon": "https://slack.github.com/static/img/favicon-neutral.png", + "color": "24292f", + "bot_id": "B01VA0FB340", + "app_unfurl_url": "https://github.com/OpenLineage/OpenLineage/blob/main/integration/spark/app/src/main/java/io/openlineage/spark/agent/EventEmitter.java#L63", + "is_app_unfurl": true, + "app_id": "A01BP7R4KNY", + "fallback": "", + "text": "```\n log.debug(\n```", + "title": "", + "footer": "", + "mrkdwn_in": [ + "text" + ] + } + ], + "thread_ts": "1693823945.734419", + "parent_user_id": "U04EZ2LPDV4" + }, + { + "client_msg_id": "b9bc6be7-8245-46fc-9dc6-28d544ce0448", + "type": "message", + "text": "Recently, we've closed that which prevents generating events for `\n```createOrReplaceTempView```\n", + "user": "U02MK6YNAQ5", + "ts": "1693899216.117129", + "blocks": [ + { + "type": "rich_text", + "block_id": "hA5c", + "elements": [ + { + "type": "rich_text_section", + "elements": [ + { + "type": "text", + "text": "Recently, we've closed that " + }, + { + "type": "link", + "url": "https://github.com/OpenLineage/OpenLineage/issues/1982" + }, + { + "type": "text", + "text": " which prevents generating events for `\n" + } + ] + }, + { + "type": "rich_text_preformatted", + "elements": [ + { + "type": "text", + "text": "createOrReplaceTempView" + } + ], + "border": 0 + }, + { + "type": "rich_text_section", + "elements": [] + } + ] + } + ], + "team": "T01CWUYP5AR", + "attachments": [ + { + "id": 1, + "footer_icon": "https://slack.github.com/static/img/favicon-neutral.png", + "ts": 1689841378, + "color": "cb2431", + "bot_id": "B01VA0FB340", + "app_unfurl_url": "https://github.com/OpenLineage/OpenLineage/issues/1982", + "is_app_unfurl": true, + "app_id": "A01BP7R4KNY", + "fallback": "#1982 [Spark] CreateViewCommand", + "text": "`CreateViewCommand` does not contain is a spark action triggered with:\n\n```\nwords.createOrReplaceTempView('words')\n```\n\nIt should be filtered in order not to generate Openlineag events, as they do have no inputs or outputs.", + "title": "#1982 [Spark] CreateViewCommand", + "title_link": "https://github.com/OpenLineage/OpenLineage/issues/1982", + "footer": "", + "fields": [ + { + "value": "", + "title": "Assignees", + "short": true + }, + { + "value": "integration/spark", + "title": "Labels", + "short": true + } + ], + "mrkdwn_in": [ + "text" + ] + } + ], + "thread_ts": "1693823945.734419", + "parent_user_id": "U04EZ2LPDV4" + }, + { + "client_msg_id": "6dfd3f2b-c88f-4b48-aaac-99351d7dddc2", + "type": "message", + "text": "and this is the code change: ", + "user": "U02MK6YNAQ5", + "ts": "1693899312.200049", + "blocks": [ + { + "type": "rich_text", + "block_id": "Mzx+", + "elements": [ + { + "type": "rich_text_section", + "elements": [ + { + "type": "text", + "text": "and this is the code change: " + }, + { + "type": "link", + "url": "https://github.com/OpenLineage/OpenLineage/pull/1987/files" + } + ] + } + ] + } + ], + "team": "T01CWUYP5AR", + "thread_ts": "1693823945.734419", + "parent_user_id": "U04EZ2LPDV4" + }, + { + "client_msg_id": "6ec9440f-ecb7-4302-adc0-bf65bd939af4", + "type": "message", + "text": "Hmm I'm a little confused here. I thought we are only filtering out events for certain specific commands, like show table etc. because its noisy right? Some important commands like MergeInto or SaveIntoDataSource used to be logged before, but I notice now that its not being logged anymore...\nI'm using 0.23.0 openlineage version.", + "user": "U04EZ2LPDV4", + "ts": "1693905562.492299", + "blocks": [ + { + "type": "rich_text", + "block_id": "NQ9", + "elements": [ + { + "type": "rich_text_section", + "elements": [ + { + "type": "text", + "text": "Hmm I'm a little confused here. I thought we are only filtering out events for certain specific commands, like show table etc. because its noisy right? Some important commands like MergeInto or SaveIntoDataSource used to be logged before, but I notice now that its not being logged anymore...\nI'm using 0.23.0 openlineage version." + } + ] + } + ] + } + ], + "team": "T01CWUYP5AR", + "thread_ts": "1693823945.734419", + "parent_user_id": "U04EZ2LPDV4" + }, + { + "client_msg_id": "fe882ea2-5856-4922-b641-08988e4d42f4", + "type": "message", + "text": "yes, we do. it's just sometimes when doing a filter, we can remove too much. but SaveIntoDataSource and MergeInto should be fine, as we do check them within the tests", + "user": "U02MK6YNAQ5", + "ts": "1693907271.993249", + "blocks": [ + { + "type": "rich_text", + "block_id": "6Ihi", + "elements": [ + { + "type": "rich_text_section", + "elements": [ + { + "type": "text", + "text": "yes, we do. it's just sometimes when doing a filter, we can remove too much. but SaveIntoDataSource and MergeInto should be fine, as we do check them within the tests" + } + ] + } + ] + } + ], + "team": "T01CWUYP5AR", + "thread_ts": "1693823945.734419", + "parent_user_id": "U04EZ2LPDV4" + } + ] + }, + { + "client_msg_id": "a9c658ee-20e7-4918-b081-db806e488787", + "type": "message", + "text": "Hi guys, I'd like to capture the `spark.databricks.clusterUsageTags.clusterAllTags` property from databricks. However, the value of this is a list of keys, and therefore cannot be supported by custom environment facet builder.\nI was thinking that capturing this property might be useful for most databricks workloads, and whether it might make sense to auto-capture it along with other databricks variables, similar to how we capture mount points for the databricks jobs.\nDoes this sound okay? If so, then I can help to contribute this functionality", + "user": "U04EZ2LPDV4", + "ts": "1693813108.356499", + "blocks": [ + { + "type": "rich_text", + "block_id": "LHT/", + "elements": [ + { + "type": "rich_text_section", + "elements": [ + { + "type": "text", + "text": "Hi guys, I'd like to capture the " + }, + { + "type": "text", + "text": "spark.databricks.clusterUsageTags.clusterAllTags", + "style": { + "code": true + } + }, + { + "type": "text", + "text": " property from databricks. However, the value of this is a list of keys, and therefore cannot be supported by custom environment facet builder.\nI was thinking that capturing this property might be useful for most databricks workloads, and whether it might make sense to auto-capture it along with other databricks variables, similar to how we capture mount points for the databricks jobs.\nDoes this sound okay? If so, then I can help to contribute this functionality" + } + ] + } + ] + } + ], + "team": "T01CWUYP5AR", + "edited": { + "user": "U04EZ2LPDV4", + "ts": "1693823112.000000" + }, + "thread_ts": "1693813108.356499", + "reply_count": 2, + "reply_users_count": 2, + "latest_reply": "1694423703.899839", + "reply_users": [ + "U01RA9B5GG2", + "U04EZ2LPDV4" + ], + "is_locked": false, + "subscribed": false, + "replies": [ + { + "client_msg_id": "c8fdc5de-b404-4811-82b2-9cd25b2196e1", + "type": "message", + "text": "Sounds good to me", + "user": "U01RA9B5GG2", + "ts": "1693824227.481319", + "blocks": [ + { + "type": "rich_text", + "block_id": "ullD6", + "elements": [ + { + "type": "rich_text_section", + "elements": [ + { + "type": "text", + "text": "Sounds good to me" + } + ] + } + ] + } + ], + "team": "T01CWUYP5AR", + "thread_ts": "1693813108.356499", + "parent_user_id": "U04EZ2LPDV4" + }, + { + "client_msg_id": "84d28e31-97b0-42b7-9f83-d855803620e9", + "type": "message", + "text": "Added this here: ", + "user": "U04EZ2LPDV4", + "ts": "1694423703.899839", + "blocks": [ + { + "type": "rich_text", + "block_id": "40oFa", + "elements": [ + { + "type": "rich_text_section", + "elements": [ + { + "type": "text", + "text": "Added this here: " + }, + { + "type": "link", + "url": "https://github.com/OpenLineage/OpenLineage/pull/2099" + } + ] + } + ] + } + ], + "team": "T01CWUYP5AR", + "attachments": [ + { + "id": 1, + "footer_icon": "https://slack.github.com/static/img/favicon-neutral.png", + "ts": 1694423544, + "color": "36a64f", + "bot_id": "B01VA0FB340", + "app_unfurl_url": "https://github.com/OpenLineage/OpenLineage/pull/2099", + "is_app_unfurl": true, + "app_id": "A01BP7R4KNY", + "fallback": "#2099 Capture clusterAllTags variable from databricks", + "text": "*Problem*\n\nAuto-collect spark.databricks.clusterUsageTags.clusterAllTags environment variable from databricks\n\nCloses: \n\n*Solution*\n\nPlease describe your change as it relates to the problem, or bug fix, as well as any dependencies. If your change requires a schema change, please describe the schema modification(s) and whether it's a _backwards-incompatible_ or _backwards-compatible_ change, then select one of the following:\n\n> *Note:* All schema changes require discussion. Please for context.\n\n☐ Your change modifies the OpenLineage model\n☐ Your change modifies one or more OpenLineage \n\nIf you're contributing a new integration, please specify the scope of the integration and how/where it has been tested (e.g., Apache Spark integration supports `S3` and `GCS` filesystem operations, tested with AWS EMR).\n\n*One-line summary:*\n*Checklist*\n\n☑︎ You've your work\n☐ Your pull request title follows our \n☐ Your changes are accompanied by tests (_if relevant_)\n☐ Your change contains a and is self-contained\n☐ You've updated any relevant documentation (_if relevant_)\n☐ Your comment includes a one-liner for the changelog about the specific purpose of the change (_if necessary_)\n☐ You've versioned the core OpenLineage model or facets according to (_if relevant_)\n☐ You've added a to source files (_if relevant_)\n\n* * *\n\nSPDX-License-Identifier: Apache-2.0 \nCopyright 2018-2023 contributors to the OpenLineage project", + "title": "#2099 Capture clusterAllTags variable from databricks", + "title_link": "https://github.com/OpenLineage/OpenLineage/pull/2099", + "footer": "", + "fields": [ + { + "value": "integration/spark", + "title": "Labels", + "short": true + } + ], + "mrkdwn_in": [ + "text" + ] + } + ], + "thread_ts": "1693813108.356499", + "parent_user_id": "U04EZ2LPDV4" + } + ] + }, + { + "client_msg_id": "c7d465d5-0fda-4602-9b45-f8f499222a61", + "type": "message", + "text": "\nThe is out now! Please to get it directly in your inbox each month.", + "user": "U02LXF3HUN7", + "ts": "1693602981.025489", + "blocks": [ + { + "type": "rich_text", + "block_id": "WNVO", + "elements": [ + { + "type": "rich_text_section", + "elements": [ + { + "type": "broadcast", + "range": "channel" + }, + { + "type": "text", + "text": "\nThe " + }, + { + "type": "link", + "url": "https://mailchi.mp/ba1d4031cbe0/openlineage-news-july-9586289?e=ef0563a7f8", + "text": "latest issue of OpenLineage News" + }, + { + "type": "text", + "text": " is out now! Please " + }, + { + "type": "link", + "url": "http://bit.ly/OL_news", + "text": "subscribe" + }, + { + "type": "text", + "text": " to get it directly in your inbox each month." + } + ] + } + ] + } + ], + "team": "T01CWUYP5AR", + "attachments": [ + { + "from_url": "http://bit.ly/OL_news", + "id": 1, + "original_url": "http://bit.ly/OL_news", + "fallback": "OpenLineage Project", + "text": "OpenLineage Project Email Forms", + "title": "OpenLineage Project", + "title_link": "http://bit.ly/OL_news", + "service_name": "apache.us14.list-manage.com" + } + ], + "reactions": [ + { + "name": "raised_hands", + "users": [ + "U02S6F54MAB", + "U01RA9B5GG2" + ], + "count": 2 + }, + { + "name": "raised_hands::skin-tone-3", + "users": [ + "U05HFGKEYVB" + ], + "count": 1 + } + ] + }, + { + "client_msg_id": "e74d588f-4242-44ea-b401-addf7e46d3f2", + "type": "message", + "text": "It sounds like there have been a few announcements at Google Next:\n\n", + "user": "U01DCLP0GU9", + "ts": "1693519820.292119", + "blocks": [ + { + "type": "rich_text", + "block_id": "j3xY", + "elements": [ + { + "type": "rich_text_section", + "elements": [ + { + "type": "text", + "text": "It sounds like there have been a few announcements at Google Next:\n" + }, + { + "type": "link", + "url": "https://cloud.google.com/data-catalog/docs/how-to/open-lineage" + }, + { + "type": "text", + "text": "\n" + }, + { + "type": "link", + "url": "https://cloud.google.com/dataproc/docs/guides/lineage" + } + ] + } + ] + } + ], + "team": "T01CWUYP5AR", + "attachments": [ + { + "image_url": "https://cloud.google.com/_static/cloud/images/social-icon-google-cloud-1200-630.png", + "image_width": 1200, + "image_height": 630, + "image_bytes": 17405, + "from_url": "https://cloud.google.com/data-catalog/docs/how-to/open-lineage", + "service_icon": "https://www.gstatic.com/devrel-devsite/prod/vbad4fd6eb290ad214822e7a397f826be8dbcc36ca2a922ba48f41fb14286829c/cloud/images/favicons/onecloud/super_cloud.png", + "id": 1, + "original_url": "https://cloud.google.com/data-catalog/docs/how-to/open-lineage", + "fallback": "Google Cloud: Integrate with OpenLineage  |  Data Catalog Documentation  |  Google Cloud", + "title": "Integrate with OpenLineage  |  Data Catalog Documentation  |  Google Cloud", + "title_link": "https://cloud.google.com/data-catalog/docs/how-to/open-lineage", + "service_name": "Google Cloud" + }, + { + "image_url": "https://cloud.google.com/_static/cloud/images/social-icon-google-cloud-1200-630.png", + "image_width": 1200, + "image_height": 630, + "image_bytes": 17405, + "from_url": "https://cloud.google.com/dataproc/docs/guides/lineage", + "service_icon": "https://www.gstatic.com/devrel-devsite/prod/vbad4fd6eb290ad214822e7a397f826be8dbcc36ca2a922ba48f41fb14286829c/cloud/images/favicons/onecloud/super_cloud.png", + "id": 2, + "original_url": "https://cloud.google.com/dataproc/docs/guides/lineage", + "fallback": "Google Cloud: Use data lineage in Dataproc  |  Dataproc Documentation  |  Google Cloud", + "title": "Use data lineage in Dataproc  |  Dataproc Documentation  |  Google Cloud", + "title_link": "https://cloud.google.com/dataproc/docs/guides/lineage", + "service_name": "Google Cloud" + } + ], + "thread_ts": "1693519820.292119", + "reply_count": 1, + "reply_users_count": 1, + "latest_reply": "1693624195.600379", + "reply_users": [ + "U01DCLP0GU9" + ], + "is_locked": false, + "subscribed": false, + "reactions": [ + { + "name": "tada", + "users": [ + "U01HNKK4XAM", + "U01DCMDFHBK", + "U05Q3HT6PBR", + "U05KKM07PJP", + "U01RA9B5GG2", + "U02MK6YNAQ5", + "U0323HG8C8H", + "U053LLVTHRN", + "U02LXF3HUN7", + "U02S6F54MAB", + "U05EN2CKBS8", + "U01DPTNCGU8", + "U05QTSH1UQP" + ], + "count": 13 + }, + { + "name": "raised_hands", + "users": [ + "U01HNKK4XAM", + "U01DCMDFHBK", + "U01HVNU6A4C", + "U05KKM07PJP", + "U01RA9B5GG2", + "U02MK6YNAQ5", + "U03DEPL699B", + "U0323HG8C8H", + "U053LLVTHRN", + "U02LXF3HUN7" + ], + "count": 10 + }, + { + "name": "heart", + "users": [ + "U01DCMDFHBK", + "U01RA9B5GG2", + "U05NMJ0NBUK", + "U053LLVTHRN", + "U02LXF3HUN7" + ], + "count": 5 + } + ], + "replies": [ + { + "client_msg_id": "1aaaa7b5-fbb8-4705-8602-2d6080b059b5", + "type": "message", + "text": "", + "user": "U01DCLP0GU9", + "ts": "1693624195.600379", + "blocks": [ + { + "type": "rich_text", + "block_id": "xTV", + "elements": [ + { + "type": "rich_text_section", + "elements": [ + { + "type": "link", + "url": "https://www.youtube.com/watch?v=zvCdrNJsxBo&t=2260s" + } + ] + } + ] + } + ], + "team": "T01CWUYP5AR", + "attachments": [ + { + "from_url": "https://www.youtube.com/watch?v=zvCdrNJsxBo&t=2260s", + "service_icon": "https://a.slack-edge.com/80588/img/unfurl_icons/youtube.png", + "thumb_url": "https://i.ytimg.com/vi/zvCdrNJsxBo/hqdefault.jpg", + "thumb_width": 480, + "thumb_height": 360, + "video_html": "", + "video_html_width": 400, + "video_html_height": 225, + "id": 1, + "original_url": "https://www.youtube.com/watch?v=zvCdrNJsxBo&t=2260s", + "fallback": "YouTube Video: What’s new in data governance", + "title": "What’s new in data governance", + "title_link": "https://www.youtube.com/watch?v=zvCdrNJsxBo&t=2260s", + "author_name": "Google Cloud", + "author_link": "https://www.youtube.com/@googlecloud", + "service_name": "YouTube", + "service_url": "https://www.youtube.com/" + } + ], + "thread_ts": "1693519820.292119", + "parent_user_id": "U01DCLP0GU9" + } + ] + }, + { + "client_msg_id": "a684bc56-75b6-4439-8234-42364c3c0c01", + "type": "message", + "text": "Will the August meeting be put up at soon? (usually it’s up in a few days :slightly_smiling_face:", + "user": "U0323HG8C8H", + "ts": "1693510399.153829", + "blocks": [ + { + "type": "rich_text", + "block_id": "s/M", + "elements": [ + { + "type": "rich_text_section", + "elements": [ + { + "type": "text", + "text": "Will the August meeting be put up at " + }, + { + "type": "link", + "url": "https://wiki.lfaidata.foundation/display/OpenLineage/Monthly+TSC+meeting" + }, + { + "type": "text", + "text": " soon? (usually it’s up in a few days " + }, + { + "type": "emoji", + "name": "slightly_smiling_face", + "unicode": "1f642" + } + ] + } + ] + } + ], + "team": "T01CWUYP5AR", + "thread_ts": "1693510399.153829", + "reply_count": 2, + "reply_users_count": 2, + "latest_reply": "1693602812.536779", + "reply_users": [ + "U01RA9B5GG2", + "U02LXF3HUN7" + ], + "is_locked": false, + "subscribed": true, + "last_read": "1693602812.536779", + "replies": [ + { + "client_msg_id": "9b157e0c-cf3b-40ba-9d95-1e3464256e93", + "type": "message", + "text": "<@U02LXF3HUN7>", + "user": "U01RA9B5GG2", + "ts": "1693562453.099679", + "blocks": [ + { + "type": "rich_text", + "block_id": "hxiDL", + "elements": [ + { + "type": "rich_text_section", + "elements": [ + { + "type": "user", + "user_id": "U02LXF3HUN7" + } + ] + } + ] + } + ], + "team": "T01CWUYP5AR", + "thread_ts": "1693510399.153829", + "parent_user_id": "U0323HG8C8H" + }, + { + "client_msg_id": "df0a8bef-5b99-4248-a47b-e52fd2a18189", + "type": "message", + "text": "The recording is on the youtube channel . I’ll update the wiki ASAP", + "user": "U02LXF3HUN7", + "ts": "1693602812.536779", + "blocks": [ + { + "type": "rich_text", + "block_id": "pKLvP", + "elements": [ + { + "type": "rich_text_section", + "elements": [ + { + "type": "text", + "text": "The recording is on the youtube channel " + }, + { + "type": "link", + "url": "https://youtu.be/0Q5dWHvIDLo", + "text": "here" + }, + { + "type": "text", + "text": ". I’ll update the wiki ASAP" + } + ] + } + ] + } + ], + "team": "T01CWUYP5AR", + "attachments": [ + { + "from_url": "https://youtu.be/0Q5dWHvIDLo", + "service_icon": "https://a.slack-edge.com/80588/img/unfurl_icons/youtube.png", + "thumb_url": "https://i.ytimg.com/vi/0Q5dWHvIDLo/hqdefault.jpg", + "thumb_width": 480, + "thumb_height": 360, + "video_html": "", + "video_html_width": 400, + "video_html_height": 225, + "id": 1, + "original_url": "https://youtu.be/0Q5dWHvIDLo", + "fallback": "YouTube Video: OpenLineage Community Meeting | August 10, 2023", + "title": "OpenLineage Community Meeting | August 10, 2023", + "title_link": "https://youtu.be/0Q5dWHvIDLo", + "author_name": "OpenLineage Project", + "author_link": "https://www.youtube.com/@openlineageproject6897", + "service_name": "YouTube", + "service_url": "https://www.youtube.com/" + } + ], + "thread_ts": "1693510399.153829", + "parent_user_id": "U0323HG8C8H", + "reactions": [ + { + "name": "white_check_mark", + "users": [ + "U0323HG8C8H" + ], + "count": 1 + } + ] + } + ] + }, + { + "client_msg_id": "74bd2622-054d-4162-b75f-1fec491ca542", + "type": "message", + "text": "ok,it`s my first use thie lineage tool. first,I added dependences in my pom.xml like this:\n<dependency>\n <groupId>io.openlineage</groupId>\n <artifactId>openlineage-java</artifactId>\n <version>0.12.0</version>\n </dependency>\n <dependency>\n <groupId>org.apache.logging.log4j</groupId>\n <artifactId>log4j-api</artifactId>\n <version>2.7</version>\n </dependency>\n <dependency>\n <groupId>org.apache.logging.log4j</groupId>\n <artifactId>log4j-core</artifactId>\n <version>2.7</version>\n </dependency>\n <dependency>\n <groupId>org.apache.logging.log4j</groupId>\n <artifactId>log4j-slf4j-impl</artifactId>\n <version>2.7</version>\n </dependency>\n <dependency>\n <groupId>io.openlineage</groupId>\n <artifactId>openlineage-spark</artifactId>\n <version>0.30.1</version>\n </dependency>\n\nmy spark version is 3.3.1 and the version can not change\n\nsecond, in file Openlineage/intergration/spark I enter command : docker-compose up and follow the steps in this doc:\n\nthere is no erro when i use notebook to execute pyspark for openlineage and I could get json message.\nbut after I enter \"docker-compose up\" ,I want to use my Idea tool to execute scala code like above,the erro happend like above. It seems that I does not configure the environment correctly. so how can i fix the problem .", + "user": "U05NGJ8AM8X", + "ts": "1693468312.450209", + "blocks": [ + { + "type": "rich_text", + "block_id": "5TpvU", + "elements": [ + { + "type": "rich_text_section", + "elements": [ + { + "type": "text", + "text": "ok,it`s my first use thie lineage tool. first,I added dependences in my pom.xml like this:\n\n io.openlineage\n openlineage-java\n 0.12.0\n \n \n org.apache.logging.log4j\n log4j-api\n 2.7\n \n \n org.apache.logging.log4j\n log4j-core\n 2.7\n \n \n org.apache.logging.log4j\n log4j-slf4j-impl\n 2.7\n \n \n io.openlineage\n openlineage-spark\n 0.30.1\n \n\nmy spark version is 3.3.1 and the version can not change\n\nsecond, in file Openlineage/intergration/spark I enter command : docker-compose up and follow the steps in this doc:\n" + }, + { + "type": "link", + "url": "https://openlineage.io/docs/integrations/spark/quickstart_local" + }, + { + "type": "text", + "text": "\nthere is no erro when i use notebook to execute pyspark for openlineage and I could get json message.\nbut after I enter \"docker-compose up\" ,I want to use my Idea tool to execute scala code like above,the erro happend like above. It seems that I does not configure the environment correctly. so how can i fix the problem ." + } + ] + } + ] + } + ], + "team": "T01CWUYP5AR", + "attachments": [ + { + "from_url": "https://openlineage.io/docs/integrations/spark/quickstart_local", + "service_icon": "https://openlineage.io/img/favicon.ico", + "id": 1, + "original_url": "https://openlineage.io/docs/integrations/spark/quickstart_local", + "fallback": "Quickstart with Jupyter | OpenLineage", + "text": "Trying out the Spark integration is super easy if you already have Docker Desktop and git installed.", + "title": "Quickstart with Jupyter | OpenLineage", + "title_link": "https://openlineage.io/docs/integrations/spark/quickstart_local", + "service_name": "openlineage.io" + } + ], + "thread_ts": "1693468312.450209", + "reply_count": 1, + "reply_users_count": 1, + "latest_reply": "1693559728.904199", + "reply_users": [ + "U02MK6YNAQ5" + ], + "is_locked": false, + "subscribed": false, + "replies": [ + { + "client_msg_id": "40ce1b5e-6c22-4196-8167-fc30863316ca", + "type": "message", + "text": "please use latest `io.openlineage:openlineage-spark:1.1.0` instead. `openlineage-java` is already contained in the jar, no need to add it on your own.", + "user": "U02MK6YNAQ5", + "ts": "1693559728.904199", + "blocks": [ + { + "type": "rich_text", + "block_id": "CeD7M", + "elements": [ + { + "type": "rich_text_section", + "elements": [ + { + "type": "text", + "text": "please use latest " + }, + { + "type": "text", + "text": "io.openlineage:openlineage-spark:1.1.0", + "style": { + "code": true + } + }, + { + "type": "text", + "text": " instead. " + }, + { + "type": "text", + "text": "openlineage-java", + "style": { + "code": true + } + }, + { + "type": "text", + "text": " is already contained in the jar, no need to add it on your own." + } + ] + } + ] + } + ], + "team": "T01CWUYP5AR", + "thread_ts": "1693468312.450209", + "parent_user_id": "U05NGJ8AM8X" + } + ] + }, + { + "client_msg_id": "cc9f0572-fc1b-4324-9e0f-d65ee769768c", + "type": "message", + "text": "hello,everyone,i can run openLineage spark code in my notebook with python,but when use my idea to execute scala code like this:\nimport org.apache.spark.internal.Logging\nimport org.apache.spark.sql.SparkSession\nimport io.openlineage.client.OpenLineageClientUtils.loadOpenLineageYaml\nimport org.apache.spark.scheduler.{SparkListener, SparkListenerApplicationEnd, SparkListenerApplicationStart}\nimport sun.java2d.marlin.MarlinUtils.logInfo\nobject Test {\n def main(args: Array[String]): Unit = {\n\n val spark = SparkSession\n .builder()\n .master(\"local\")\n .appName(\"test\")\n .config(\"spark.jars.packages\",\"io.openlineage:openlineage-spark:0.12.0\")\n .config(\"spark.extraListeners\",\"io.openlineage.spark.agent.OpenLineageSparkListener\")\n .config(\"spark.openlineage.transport.type\",\"console\")\n .getOrCreate()\n\n spark.sparkContext.setLogLevel(\"INFO\")\n\n //spark.sparkContext.addSparkListener(new MySparkAppListener)\n import spark.implicits._\n val input = Seq((1, \"zs\", 2020), (2, \"ls\", 2023)).toDF(\"id\", \"name\", \"year\")\n\n input.select(\"id\", \"name\").orderBy(\"id\").show()\n\n }\n\n}\n\nthere is something wrong:\nException in thread \"spark-listener-group-shared\" java.lang.NoSuchMethodError: io.openlineage.client.OpenLineageClientUtils.loadOpenLineageYaml(Ljava/io/InputStream;)Lio/openlineage/client/OpenLineageYaml;\n\tat io.openlineage.spark.agent.ArgumentParser.extractOpenlineageConfFromSparkConf(ArgumentParser.java:114)\n\tat io.openlineage.spark.agent.ArgumentParser.parse(ArgumentParser.java:78)\n\tat io.openlineage.spark.agent.OpenLineageSparkListener.initializeContextFactoryIfNotInitialized(OpenLineageSparkListener.java:277)\n\tat io.openlineage.spark.agent.OpenLineageSparkListener.onApplicationStart(OpenLineageSparkListener.java:267)\n\tat org.apache.spark.scheduler.SparkListenerBus.doPostEvent(SparkListenerBus.scala:55)\n\tat org.apache.spark.scheduler.SparkListenerBus.doPostEvent$(SparkListenerBus.scala:28)\n\tat org.apache.spark.scheduler.AsyncEventQueue.doPostEvent(AsyncEventQueue.scala:37)\n\tat org.apache.spark.scheduler.AsyncEventQueue.doPostEvent(AsyncEventQueue.scala:37)\n\tat org.apache.spark.util.ListenerBus.postToAll(ListenerBus.scala:117)\n\tat org.apache.spark.util.ListenerBus.postToAll$(ListenerBus.scala:101)\n\tat org.apache.spark.scheduler.AsyncEventQueue.super$postToAll(AsyncEventQueue.scala:105)\n\tat org.apache.spark.scheduler.AsyncEventQueue.$anonfun$dispatch$1(AsyncEventQueue.scala:105)\n\tat scala.runtime.java8.JFunction0$mcJ$sp.apply(JFunction0$mcJ$sp.java:23)\n\tat scala.util.DynamicVariable.withValue(DynamicVariable.scala:62)\n\tat $apache$spark$scheduler$AsyncEventQueue$$dispatch(AsyncEventQueue.scala:100)\n\tat org.apache.spark.scheduler.AsyncEventQueue$$anon$2.$anonfun$run$1(AsyncEventQueue.scala:96)\n\tat org.apache.spark.util.Utils$.tryOrStopSparkContext(Utils.scala:1446)\n\tat org.apache.spark.scheduler.AsyncEventQueue$$anon$2.run(AsyncEventQueue.scala:96)\n\ni want to know how can i set idea scala environment correctly", + "user": "U05NGJ8AM8X", + "ts": "1693463508.522729", + "blocks": [ + { + "type": "rich_text", + "block_id": "OyUng", + "elements": [ + { + "type": "rich_text_section", + "elements": [ + { + "type": "text", + "text": "hello,everyone,i can run openLineage spark code in my notebook with python,but when use my idea to execute scala code like this:\nimport org.apache.spark.internal.Logging\nimport org.apache.spark.sql.SparkSession\nimport io.openlineage.client.OpenLineageClientUtils.loadOpenLineageYaml\nimport org.apache.spark.scheduler.{SparkListener, SparkListenerApplicationEnd, SparkListenerApplicationStart}\nimport sun.java2d.marlin.MarlinUtils.logInfo\nobject Test {\n def main(args: Array[String]): Unit = {\n\n val spark = SparkSession\n .builder()\n .master(\"local\")\n .appName(\"test\")\n .config(\"spark.jars.packages\",\"io.openlineage:openlineage-spark:0.12.0\")\n .config(\"spark.extraListeners\",\"io.openlineage.spark.agent.OpenLineageSparkListener\")\n .config(\"spark.openlineage.transport.type\",\"console\")\n .getOrCreate()\n\n spark.sparkContext.setLogLevel(\"INFO\")\n\n //spark.sparkContext.addSparkListener(new MySparkAppListener)\n import spark.implicits._\n val input = Seq((1, \"zs\", 2020), (2, \"ls\", 2023)).toDF(\"id\", \"name\", \"year\")\n\n input.select(\"id\", \"name\").orderBy(\"id\").show()\n\n }\n\n}\n\nthere is something wrong:\nException in thread \"spark-listener-group-shared\" java.lang.NoSuchMethodError: io.openlineage.client.OpenLineageClientUtils.loadOpenLineageYaml(Ljava/io/InputStream;)Lio/openlineage/client/OpenLineageYaml;\n\tat io.openlineage.spark.agent.ArgumentParser.extractOpenlineageConfFromSparkConf(ArgumentParser.java:114)\n\tat io.openlineage.spark.agent.ArgumentParser.parse(ArgumentParser.java:78)\n\tat io.openlineage.spark.agent.OpenLineageSparkListener.initializeContextFactoryIfNotInitialized(OpenLineageSparkListener.java:277)\n\tat io.openlineage.spark.agent.OpenLineageSparkListener.onApplicationStart(OpenLineageSparkListener.java:267)\n\tat org.apache.spark.scheduler.SparkListenerBus.doPostEvent(SparkListenerBus.scala:55)\n\tat org.apache.spark.scheduler.SparkListenerBus.doPostEvent$(SparkListenerBus.scala:28)\n\tat org.apache.spark.scheduler.AsyncEventQueue.doPostEvent(AsyncEventQueue.scala:37)\n\tat org.apache.spark.scheduler.AsyncEventQueue.doPostEvent(AsyncEventQueue.scala:37)\n\tat org.apache.spark.util.ListenerBus.postToAll(ListenerBus.scala:117)\n\tat org.apache.spark.util.ListenerBus.postToAll$(ListenerBus.scala:101)\n\tat org.apache.spark.scheduler.AsyncEventQueue.super$postToAll(AsyncEventQueue.scala:105)\n\tat org.apache.spark.scheduler.AsyncEventQueue.$anonfun$dispatch$1(AsyncEventQueue.scala:105)\n\tat scala.runtime.java8.JFunction0$mcJ$sp.apply(JFunction0$mcJ$sp.java:23)\n\tat scala.util.DynamicVariable.withValue(DynamicVariable.scala:62)\n\tat " + }, + { + "type": "link", + "url": "http://org.apache.spark.scheduler.AsyncEventQueue.org", + "text": "org.apache.spark.scheduler.AsyncEventQueue.org" + }, + { + "type": "text", + "text": "$apache$spark$scheduler$AsyncEventQueue$$dispatch(AsyncEventQueue.scala:100)\n\tat org.apache.spark.scheduler.AsyncEventQueue$$anon$2.$anonfun$run$1(AsyncEventQueue.scala:96)\n\tat org.apache.spark.util.Utils$.tryOrStopSparkContext(Utils.scala:1446)\n\tat org.apache.spark.scheduler.AsyncEventQueue$$anon$2.run(AsyncEventQueue.scala:96)\n\ni want to know how can i set idea scala environment correctly" + } + ] + } + ] + } + ], + "team": "T01CWUYP5AR", + "thread_ts": "1693463508.522729", + "reply_count": 1, + "reply_users_count": 1, + "latest_reply": "1693465121.120999", + "reply_users": [ + "U02MK6YNAQ5" + ], + "is_locked": false, + "subscribed": false, + "replies": [ + { + "client_msg_id": "7aa5f249-2197-4c0b-aaa8-0e2053f62c03", + "type": "message", + "text": "`io.openlineage:openlineage-spark:0.12.0` -> could you repeat the steps with newer version?", + "user": "U02MK6YNAQ5", + "ts": "1693465121.120999", + "blocks": [ + { + "type": "rich_text", + "block_id": "nz3", + "elements": [ + { + "type": "rich_text_section", + "elements": [ + { + "type": "text", + "text": "io.openlineage:openlineage-spark:0.12.0", + "style": { + "code": true + } + }, + { + "type": "text", + "text": " -> could you repeat the steps with newer version?" + } + ] + } + ] + } + ], + "team": "T01CWUYP5AR", + "thread_ts": "1693463508.522729", + "parent_user_id": "U05NGJ8AM8X" + } + ] + }, + { + "client_msg_id": "077333c6-55c2-4dd0-b529-25220920e053", + "type": "message", + "text": "Can anyone let 3 people stuck downstairs into the 7th floor?", + "user": "U05EC8WB74N", + "ts": "1693445911.744069", + "blocks": [ + { + "type": "rich_text", + "block_id": "nkv", + "elements": [ + { + "type": "rich_text_section", + "elements": [ + { + "type": "text", + "text": "Can anyone let 3 people stuck downstairs into the 7th floor?" + } + ] + } + ] + } + ], + "team": "T01CWUYP5AR", + "thread_ts": "1693445911.744069", + "reply_count": 1, + "reply_users_count": 1, + "latest_reply": "1693452321.429109", + "reply_users": [ + "U01DCMDFHBK" + ], + "is_locked": false, + "subscribed": false, + "reactions": [ + { + "name": "+1", + "users": [ + "U01DCMDFHBK" + ], + "count": 1 + } + ], + "replies": [ + { + "client_msg_id": "4634016F-F84A-465A-AF4B-4D0E2B5743DE", + "type": "message", + "text": "Sorry about that!", + "user": "U01DCMDFHBK", + "ts": "1693452321.429109", + "blocks": [ + { + "type": "rich_text", + "block_id": "n+oGh", + "elements": [ + { + "type": "rich_text_section", + "elements": [ + { + "type": "text", + "text": "S" + }, + { + "type": "text", + "text": "orry about that!" + } + ] + } + ] + } + ], + "team": "T01CWUYP5AR", + "thread_ts": "1693445911.744069", + "parent_user_id": "U05EC8WB74N" + } + ] + }, + { + "client_msg_id": "8550eec8-1a37-46ed-b901-92af92360a0f", + "type": "message", + "text": "\nFriendly reminder: there’s a meetup at Astronomer’s offices in SF!", + "user": "U02LXF3HUN7", + "ts": "1693410605.894959", + "blocks": [ + { + "type": "rich_text", + "block_id": "rdN5X", + "elements": [ + { + "type": "rich_text_section", + "elements": [ + { + "type": "broadcast", + "range": "channel" + }, + { + "type": "text", + "text": "\nFriendly reminder: there’s a meetup " + }, + { + "type": "link", + "url": "https://openlineage.slack.com/archives/C01CK9T7HKR/p1692973763570629", + "text": "tonight" + }, + { + "type": "text", + "text": " at Astronomer’s offices in SF!" + } + ] + } + ] + } + ], + "team": "T01CWUYP5AR", + "attachments": [ + { + "from_url": "https://openlineage.slack.com/archives/C01CK9T7HKR/p1692973763570629", + "ts": "1692973763.570629", + "author_id": "U02LXF3HUN7", + "channel_id": "C01CK9T7HKR", + "channel_team": "T01CWUYP5AR", + "is_msg_unfurl": true, + "message_blocks": [ + { + "team": "T01CWUYP5AR", + "channel": "C01CK9T7HKR", + "ts": "1692973763.570629", + "message": { + "blocks": [ + { + "type": "rich_text", + "block_id": "T3fR", + "elements": [ + { + "type": "rich_text_section", + "elements": [ + { + "type": "broadcast", + "range": "channel" + }, + { + "type": "text", + "text": "\nFriendly reminder: our next in-person meetup is next Wednesday, August 30th in San Francisco at Astronomer’s offices in the Financial District. You can sign up and find the details on the " + }, + { + "type": "link", + "url": "https://www.meetup.com/meetup-group-bnfqymxe/events/295195280/?utm_medium=referral&utm_campaign=share-btn_savedevents_share_modal&utm_source=link", + "text": "meetup event page" + }, + { + "type": "text", + "text": "." + } + ] + } + ] + } + ] + } + } + ], + "id": 1, + "original_url": "https://openlineage.slack.com/archives/C01CK9T7HKR/p1692973763570629", + "fallback": "[August 25th, 2023 7:29 AM] michael282: \nFriendly reminder: our next in-person meetup is next Wednesday, August 30th in San Francisco at Astronomer’s offices in the Financial District. You can sign up and find the details on the .", + "text": "\nFriendly reminder: our next in-person meetup is next Wednesday, August 30th in San Francisco at Astronomer’s offices in the Financial District. You can sign up and find the details on the .", + "author_name": "Michael Robinson", + "author_link": "https://openlineage.slack.com/team/U02LXF3HUN7", + "author_icon": "https://avatars.slack-edge.com/2022-01-25/3019716733729_66fea720e9504dc08144_48.jpg", + "author_subname": "Michael Robinson", + "mrkdwn_in": [ + "text" + ], + "footer": "Slack Conversation" + } + ], + "thread_ts": "1693410605.894959", + "reply_count": 1, + "reply_users_count": 1, + "latest_reply": "1693412131.696559", + "reply_users": [ + "U01DCLP0GU9" + ], + "is_locked": false, + "subscribed": true, + "last_read": "1693412131.696559", + "reactions": [ + { + "name": "white_check_mark", + "users": [ + "U0323HG8C8H" + ], + "count": 1 + } + ], + "replies": [ + { + "client_msg_id": "7837D15A-EFC0-49C1-B259-D9099711E94F", + "type": "message", + "text": "I’ll be there and looking forward to see <@U04AZ7992SU> ‘s presentation ", + "user": "U01DCLP0GU9", + "ts": "1693412131.696559", + "blocks": [ + { + "type": "rich_text", + "block_id": "F85", + "elements": [ + { + "type": "rich_text_section", + "elements": [ + { + "type": "text", + "text": "I" + }, + { + "type": "text", + "text": "’" + }, + { + "type": "text", + "text": "ll be there and looking forward to see " + }, + { + "type": "user", + "user_id": "U04AZ7992SU" + }, + { + "type": "text", + "text": " ‘s presentation " + } + ] + } + ] + } + ], + "team": "T01CWUYP5AR", + "thread_ts": "1693410605.894959", + "parent_user_id": "U02LXF3HUN7" + } + ] + }, + { + "client_msg_id": "f42da97f-7aa0-4298-ba23-92bf84894fb9", + "type": "message", + "text": "Hi, Will really appreciate if someone can guide me or provide me any pointer - if they have been able to implement authentication/authorization for access to Marquez. Have not seen much info around it. Any pointers greatly appreciated. Thanks in advance.", + "user": "U05JY6MN8MS", + "ts": "1693397729.978309", + "blocks": [ + { + "type": "rich_text", + "block_id": "1xS", + "elements": [ + { + "type": "rich_text_section", + "elements": [ + { + "type": "text", + "text": "Hi, Will really appreciate if someone can guide me or provide me any pointer - if they have been able to implement authentication/authorization for access to Marquez. Have not seen much info around it. Any pointers greatly appreciated. Thanks in advance." + } + ] + } + ] + } + ], + "team": "T01CWUYP5AR", + "thread_ts": "1693397729.978309", + "reply_count": 1, + "reply_users_count": 1, + "latest_reply": "1693412598.034059", + "reply_users": [ + "U01DCLP0GU9" + ], + "is_locked": false, + "subscribed": false, + "replies": [ + { + "client_msg_id": "761A1B6F-8A7E-4F32-B083-BE3B49D990F2", + "type": "message", + "text": "I’ve seen people do this through the ingress controller in Kubernetes. Unfortunately I don’t have documentation besides k8s specific ones you would find for the ingress controller you’re using. You’d redirect any unauthenticated request to your identity provider ", + "user": "U01DCLP0GU9", + "ts": "1693412598.034059", + "blocks": [ + { + "type": "rich_text", + "block_id": "QR1c", + "elements": [ + { + "type": "rich_text_section", + "elements": [ + { + "type": "text", + "text": "I’ve seen people do this through the ingress controller in Kubernetes" + }, + { + "type": "text", + "text": "." + }, + { + "type": "text", + "text": " Unfortunately I " + }, + { + "type": "text", + "text": "don’t" + }, + { + "type": "text", + "text": " have documentation besides k8s specific ones you would find for the ingress controller you’re using" + }, + { + "type": "text", + "text": "." + }, + { + "type": "text", + "text": " You’d redirect any unauthenticated request to your identity provider " + } + ] + } + ] + } + ], + "team": "T01CWUYP5AR", + "thread_ts": "1693397729.978309", + "parent_user_id": "U05JY6MN8MS", + "reactions": [ + { + "name": "gratitude-thank-you", + "users": [ + "U05JY6MN8MS" + ], + "count": 1 + } + ] + } + ] + }, + { + "client_msg_id": "b40995eb-c851-4057-b01e-55c9e96334f3", + "type": "message", + "text": "for namespaces, if my data is moving between sources (SFTP -> GCS -> Azure Blob (synapse connects to parquet datasets) then should my namespace be based on the client I am working with? my current namespace has been to refer to the bucket, but that falls apart when considering the data sources and some destinations. perhaps I should just add a field for client-name instead to have a consolidated view?", + "user": "U05NMJ0NBUK", + "ts": "1693329152.193929", + "blocks": [ + { + "type": "rich_text", + "block_id": "jb/UL", + "elements": [ + { + "type": "rich_text_section", + "elements": [ + { + "type": "text", + "text": "for namespaces, if my data is moving between sources (SFTP -> GCS -> Azure Blob (synapse connects to parquet datasets) then should my namespace be based on the client I am working with? my current namespace has been " + }, + { + "type": "link", + "url": "gcs://client-name" + }, + { + "type": "text", + "text": " to refer to the bucket, but that falls apart when considering the data sources and some destinations. perhaps I should just add a field for client-name instead to have a consolidated view?" + } + ] + } + ] + } + ], + "team": "T01CWUYP5AR", + "thread_ts": "1693329152.193929", + "reply_count": 4, + "reply_users_count": 2, + "latest_reply": "1693415198.889139", + "reply_users": [ + "U01RA9B5GG2", + "U05NMJ0NBUK" + ], + "is_locked": false, + "subscribed": false, + "replies": [ + { + "client_msg_id": "8ca4e2bc-3db6-4e27-be54-8bd928a797d0", + "type": "message", + "text": "> then should my namespace be based on the client I am working with?\nI think each of those sources should be a different namespace?", + "user": "U01RA9B5GG2", + "ts": "1693407188.766089", + "blocks": [ + { + "type": "rich_text", + "block_id": "UTao", + "elements": [ + { + "type": "rich_text_quote", + "elements": [ + { + "type": "text", + "text": "then should my namespace be based on the client I am working with?" + } + ] + }, + { + "type": "rich_text_section", + "elements": [ + { + "type": "text", + "text": "\nI think each of those sources should be a different namespace?" + } + ] + } + ] + } + ], + "team": "T01CWUYP5AR", + "thread_ts": "1693329152.193929", + "parent_user_id": "U05NMJ0NBUK" + }, + { + "client_msg_id": "ddfe6454-c9c8-40c1-813a-98f67405d6ff", + "type": "message", + "text": "got it, yeah I was kind of picturing as one namespace for the client (we handle many clients but they are completely distinct entities). I was able to get it to work with multiple namespaces like you suggested and Marquez was able to plot everything correctly in the visualization", + "user": "U05NMJ0NBUK", + "ts": "1693414793.644179", + "blocks": [ + { + "type": "rich_text", + "block_id": "UwxW", + "elements": [ + { + "type": "rich_text_section", + "elements": [ + { + "type": "text", + "text": "got it, yeah I was kind of picturing as one namespace for the client (we handle many clients but they are completely distinct entities). I was able to get it to work with multiple namespaces like you suggested and Marquez was able to plot everything correctly in the visualization" + } + ] + } + ] + } + ], + "team": "T01CWUYP5AR", + "thread_ts": "1693329152.193929", + "parent_user_id": "U05NMJ0NBUK" + }, + { + "client_msg_id": "c3070060-9622-4b17-94fc-e119ada63562", + "type": "message", + "text": "I noticed some of my Dataset facets make more sense as Run facets, for example, the name of the specific file I processed and how many rows of data / size of the data for that schedule. that won't impact the Run facets Airflow provides right? I can still have the schedule information + my custom run facets?", + "user": "U05NMJ0NBUK", + "ts": "1693414878.749489", + "blocks": [ + { + "type": "rich_text", + "block_id": "H46+", + "elements": [ + { + "type": "rich_text_section", + "elements": [ + { + "type": "text", + "text": "I noticed some of my Dataset facets make more sense as Run facets, for example, the name of the specific file I processed and how many rows of data / size of the data for that schedule. that won't impact the Run facets Airflow provides right? I can still have the schedule information + my custom run facets?" + } + ] + } + ] + } + ], + "team": "T01CWUYP5AR", + "thread_ts": "1693329152.193929", + "parent_user_id": "U05NMJ0NBUK" + }, + { + "client_msg_id": "e84ab19a-7437-4fe3-9a12-27b2d7069434", + "type": "message", + "text": "Yes, unless you name it the same as one of the Airflow facets :slightly_smiling_face:", + "user": "U01RA9B5GG2", + "ts": "1693415198.889139", + "blocks": [ + { + "type": "rich_text", + "block_id": "9yt4", + "elements": [ + { + "type": "rich_text_section", + "elements": [ + { + "type": "text", + "text": "Yes, unless you name it the same as one of the Airflow facets " + }, + { + "type": "emoji", + "name": "slightly_smiling_face", + "unicode": "1f642" + } + ] + } + ] + } + ], + "team": "T01CWUYP5AR", + "thread_ts": "1693329152.193929", + "parent_user_id": "U05NMJ0NBUK" + } + ] + }, + { + "client_msg_id": "1a1fe5e1-6305-4ae6-87ab-aa7e2b732990", + "type": "message", + "text": "hi folks, for now I'm producing `.jsonl` (or `.ndjson` ) files with one event per line, do you know if there's any way to validate those? would standard JSON Schema tools work?", + "user": "U05HFGKEYVB", + "ts": "1693300839.710459", + "blocks": [ + { + "type": "rich_text", + "block_id": "Od1G", + "elements": [ + { + "type": "rich_text_section", + "elements": [ + { + "type": "text", + "text": "hi folks, for now I'm producing " + }, + { + "type": "text", + "text": ".jsonl", + "style": { + "code": true + } + }, + { + "type": "text", + "text": " (or " + }, + { + "type": "text", + "text": ".ndjson", + "style": { + "code": true + } + }, + { + "type": "text", + "text": " ) files with one event per line, do you know if there's any way to validate those? would standard JSON Schema tools work?" + } + ] + } + ] + } + ], + "team": "T01CWUYP5AR", + "thread_ts": "1693300839.710459", + "reply_count": 1, + "reply_users_count": 1, + "latest_reply": "1693321109.057039", + "reply_users": [ + "U05HFGKEYVB" + ], + "is_locked": false, + "subscribed": false, + "replies": [ + { + "client_msg_id": "e0bff8b7-7427-490d-9cb9-26dd87e2c1a5", + "type": "message", + "text": "reply by <@U0544QC1DS9>: yes :slightly_smiling_face::100:", + "user": "U05HFGKEYVB", + "ts": "1693321109.057039", + "blocks": [ + { + "type": "rich_text", + "block_id": "ofv", + "elements": [ + { + "type": "rich_text_section", + "elements": [ + { + "type": "text", + "text": "reply by " + }, + { + "type": "user", + "user_id": "U0544QC1DS9" + }, + { + "type": "text", + "text": ": yes " + }, + { + "type": "emoji", + "name": "slightly_smiling_face", + "unicode": "1f642" + }, + { + "type": "emoji", + "name": "100", + "unicode": "1f4af" + } + ] + } + ] + } + ], + "team": "T01CWUYP5AR", + "thread_ts": "1693300839.710459", + "parent_user_id": "U05HFGKEYVB", + "reactions": [ + { + "name": "+1", + "users": [ + "U01RA9B5GG2" + ], + "count": 1 + } + ] + } + ] + }, + { + "type": "message", + "text": "Hello, I'm currently in the process of following the instructions outlined in the provided getting started guide at . However, I've encountered a problem while attempting to complete **Step 1** of the guide. Unfortunately, I'm encountering an internal server error at this stage. I did manage to successfully run Marquez, but it appears that there might be an issue that needs to be addressed. I have attached screen shots.", + "files": [ + { + "id": "F05PSGC7D8E", + "created": 1693292986, + "timestamp": 1693292986, + "name": "image.png", + "title": "image.png", + "mimetype": "image/png", + "filetype": "png", + "pretty_type": "PNG", + "user": "U05QNRSQW1E", + "user_team": "T01CWUYP5AR", + "editable": false, + "size": 66903, + "mode": "hosted", + "is_external": false, + "external_type": "", + "is_public": true, + "public_url_shared": false, + "display_as_bot": false, + "username": "", + "url_private": "https://files.slack.com/files-pri/T01CWUYP5AR-F05PSGC7D8E/image.png", + "url_private_download": "https://files.slack.com/files-pri/T01CWUYP5AR-F05PSGC7D8E/download/image.png", + "media_display_type": "unknown", + "thumb_64": "https://files.slack.com/files-tmb/T01CWUYP5AR-F05PSGC7D8E-3fd1285666/image_64.png", + "thumb_80": "https://files.slack.com/files-tmb/T01CWUYP5AR-F05PSGC7D8E-3fd1285666/image_80.png", + "thumb_360": "https://files.slack.com/files-tmb/T01CWUYP5AR-F05PSGC7D8E-3fd1285666/image_360.png", + "thumb_360_w": 360, + "thumb_360_h": 181, + "thumb_480": "https://files.slack.com/files-tmb/T01CWUYP5AR-F05PSGC7D8E-3fd1285666/image_480.png", + "thumb_480_w": 480, + "thumb_480_h": 241, + "thumb_160": "https://files.slack.com/files-tmb/T01CWUYP5AR-F05PSGC7D8E-3fd1285666/image_160.png", + "thumb_720": "https://files.slack.com/files-tmb/T01CWUYP5AR-F05PSGC7D8E-3fd1285666/image_720.png", + "thumb_720_w": 720, + "thumb_720_h": 361, + "thumb_800": "https://files.slack.com/files-tmb/T01CWUYP5AR-F05PSGC7D8E-3fd1285666/image_800.png", + "thumb_800_w": 800, + "thumb_800_h": 401, + "thumb_960": "https://files.slack.com/files-tmb/T01CWUYP5AR-F05PSGC7D8E-3fd1285666/image_960.png", + "thumb_960_w": 960, + "thumb_960_h": 482, + "thumb_1024": "https://files.slack.com/files-tmb/T01CWUYP5AR-F05PSGC7D8E-3fd1285666/image_1024.png", + "thumb_1024_w": 1024, + "thumb_1024_h": 514, + "original_w": 1907, + "original_h": 957, + "thumb_tiny": "AwAYADCqLiYf8tW/OlNzN/z0cfjUIopgS/aZv+er/nR9pn/56v8AnUVFAiY3Mx6yP+dN8+X/AJ6N+dR0UAAopT1pKBhRSig0AJRRRQI//9k=", + "permalink": "https://openlineage.slack.com/files/U05QNRSQW1E/F05PSGC7D8E/image.png", + "permalink_public": "https://slack-files.com/T01CWUYP5AR-F05PSGC7D8E-683e6f3655", + "is_starred": false, + "has_rich_preview": false, + "file_access": "visible" + }, + { + "id": "F05PW99E3L5", + "created": 1693293378, + "timestamp": 1693293378, + "name": "image.png", + "title": "image.png", + "mimetype": "image/png", + "filetype": "png", + "pretty_type": "PNG", + "user": "U05QNRSQW1E", + "user_team": "T01CWUYP5AR", + "editable": false, + "size": 50611, + "mode": "hosted", + "is_external": false, + "external_type": "", + "is_public": true, + "public_url_shared": false, + "display_as_bot": false, + "username": "", + "url_private": "https://files.slack.com/files-pri/T01CWUYP5AR-F05PW99E3L5/image.png", + "url_private_download": "https://files.slack.com/files-pri/T01CWUYP5AR-F05PW99E3L5/download/image.png", + "media_display_type": "unknown", + "thumb_64": "https://files.slack.com/files-tmb/T01CWUYP5AR-F05PW99E3L5-0dd146164b/image_64.png", + "thumb_80": "https://files.slack.com/files-tmb/T01CWUYP5AR-F05PW99E3L5-0dd146164b/image_80.png", + "thumb_360": "https://files.slack.com/files-tmb/T01CWUYP5AR-F05PW99E3L5-0dd146164b/image_360.png", + "thumb_360_w": 360, + "thumb_360_h": 191, + "thumb_480": "https://files.slack.com/files-tmb/T01CWUYP5AR-F05PW99E3L5-0dd146164b/image_480.png", + "thumb_480_w": 480, + "thumb_480_h": 255, + "thumb_160": "https://files.slack.com/files-tmb/T01CWUYP5AR-F05PW99E3L5-0dd146164b/image_160.png", + "thumb_720": "https://files.slack.com/files-tmb/T01CWUYP5AR-F05PW99E3L5-0dd146164b/image_720.png", + "thumb_720_w": 720, + "thumb_720_h": 383, + "thumb_800": "https://files.slack.com/files-tmb/T01CWUYP5AR-F05PW99E3L5-0dd146164b/image_800.png", + "thumb_800_w": 800, + "thumb_800_h": 425, + "thumb_960": "https://files.slack.com/files-tmb/T01CWUYP5AR-F05PW99E3L5-0dd146164b/image_960.png", + "thumb_960_w": 960, + "thumb_960_h": 510, + "thumb_1024": "https://files.slack.com/files-tmb/T01CWUYP5AR-F05PW99E3L5-0dd146164b/image_1024.png", + "thumb_1024_w": 1024, + "thumb_1024_h": 544, + "original_w": 1048, + "original_h": 557, + "thumb_tiny": "AwAZADClk0hOaXB9KTFaEBSUtFACUUUUAJlvT9KMn0qWmt1pWKuMyfSlB9qKKLCuGaKKKYj/2Q==", + "permalink": "https://openlineage.slack.com/files/U05QNRSQW1E/F05PW99E3L5/image.png", + "permalink_public": "https://slack-files.com/T01CWUYP5AR-F05PW99E3L5-4393f52650", + "is_starred": false, + "has_rich_preview": false, + "file_access": "visible" + } + ], + "upload": false, + "user": "U05QNRSQW1E", + "ts": "1693293484.701439", + "blocks": [ + { + "type": "rich_text", + "block_id": "omnO", + "elements": [ + { + "type": "rich_text_section", + "elements": [ + { + "type": "text", + "text": "Hello, I'm currently in the process of following the instructions outlined in the provided getting started guide at " + }, + { + "type": "link", + "url": "https://openlineage.io/getting-started/" + }, + { + "type": "text", + "text": ". However, I've encountered a problem while attempting to complete **Step 1** of the guide. Unfortunately, I'm encountering an internal server error at this stage. I did manage to successfully run Marquez, but it appears that there might be an issue that needs to be addressed. I have attached screen shots." + } + ] + } + ] + } + ], + "client_msg_id": "db59bd12-2597-4c8f-85c4-a32b2603c798", + "thread_ts": "1693293484.701439", + "reply_count": 3, + "reply_users_count": 3, + "latest_reply": "1693317758.935859", + "reply_users": [ + "U02S6F54MAB", + "U05QNRSQW1E", + "U01RA9B5GG2" + ], + "is_locked": false, + "subscribed": false, + "replies": [ + { + "client_msg_id": "2ed33632-14e6-47f6-9cc6-d4df5594ab8c", + "type": "message", + "text": "is 5000 port taken by any other application? or `./docker/up.sh` has some errors in logs?", + "user": "U02S6F54MAB", + "ts": "1693293618.259409", + "blocks": [ + { + "type": "rich_text", + "block_id": "Tmd", + "elements": [ + { + "type": "rich_text_section", + "elements": [ + { + "type": "text", + "text": "is 5000 port taken by any other application? or " + }, + { + "type": "text", + "text": "./docker/up.sh", + "style": { + "code": true + } + }, + { + "type": "text", + "text": " has some errors in logs?" + } + ] + } + ] + } + ], + "team": "T01CWUYP5AR", + "thread_ts": "1693293484.701439", + "parent_user_id": "U05QNRSQW1E" + }, + { + "type": "message", + "text": "<@U02S6F54MAB> 5000 port is not taken by any other application. The logs show some errors but I am not sure what is the issue here.", + "files": [ + { + "id": "F05PKBM0RRV", + "created": 1693300976, + "timestamp": 1693300976, + "name": "image.png", + "title": "image.png", + "mimetype": "image/png", + "filetype": "png", + "pretty_type": "PNG", + "user": "U05QNRSQW1E", + "user_team": "T01CWUYP5AR", + "editable": false, + "size": 50149, + "mode": "hosted", + "is_external": false, + "external_type": "", + "is_public": true, + "public_url_shared": false, + "display_as_bot": false, + "username": "", + "url_private": "https://files.slack.com/files-pri/T01CWUYP5AR-F05PKBM0RRV/image.png", + "url_private_download": "https://files.slack.com/files-pri/T01CWUYP5AR-F05PKBM0RRV/download/image.png", + "media_display_type": "unknown", + "thumb_64": "https://files.slack.com/files-tmb/T01CWUYP5AR-F05PKBM0RRV-3396b1753a/image_64.png", + "thumb_80": "https://files.slack.com/files-tmb/T01CWUYP5AR-F05PKBM0RRV-3396b1753a/image_80.png", + "thumb_360": "https://files.slack.com/files-tmb/T01CWUYP5AR-F05PKBM0RRV-3396b1753a/image_360.png", + "thumb_360_w": 360, + "thumb_360_h": 56, + "thumb_480": "https://files.slack.com/files-tmb/T01CWUYP5AR-F05PKBM0RRV-3396b1753a/image_480.png", + "thumb_480_w": 480, + "thumb_480_h": 75, + "thumb_160": "https://files.slack.com/files-tmb/T01CWUYP5AR-F05PKBM0RRV-3396b1753a/image_160.png", + "thumb_720": "https://files.slack.com/files-tmb/T01CWUYP5AR-F05PKBM0RRV-3396b1753a/image_720.png", + "thumb_720_w": 720, + "thumb_720_h": 113, + "thumb_800": "https://files.slack.com/files-tmb/T01CWUYP5AR-F05PKBM0RRV-3396b1753a/image_800.png", + "thumb_800_w": 800, + "thumb_800_h": 125, + "thumb_960": "https://files.slack.com/files-tmb/T01CWUYP5AR-F05PKBM0RRV-3396b1753a/image_960.png", + "thumb_960_w": 960, + "thumb_960_h": 150, + "thumb_1024": "https://files.slack.com/files-tmb/T01CWUYP5AR-F05PKBM0RRV-3396b1753a/image_1024.png", + "thumb_1024_w": 1024, + "thumb_1024_h": 160, + "original_w": 1598, + "original_h": 250, + "thumb_tiny": "AwAHADCgetJ+FKaSgBOfQUEE+lLS0AMxSGnGmmgD/9k=", + "permalink": "https://openlineage.slack.com/files/U05QNRSQW1E/F05PKBM0RRV/image.png", + "permalink_public": "https://slack-files.com/T01CWUYP5AR-F05PKBM0RRV-27cc4fb62d", + "is_starred": false, + "has_rich_preview": false, + "file_access": "visible" + } + ], + "upload": false, + "user": "U05QNRSQW1E", + "display_as_bot": false, + "ts": "1693300981.357099", + "blocks": [ + { + "type": "rich_text", + "block_id": "luX", + "elements": [ + { + "type": "rich_text_section", + "elements": [ + { + "type": "user", + "user_id": "U02S6F54MAB" + }, + { + "type": "text", + "text": " 5000 port is not taken by any other application. The logs show some errors but I am not sure what is the issue here." + } + ] + } + ] + } + ], + "client_msg_id": "3dd8369b-2ee3-4b65-bcc4-93695ee8f310", + "thread_ts": "1693293484.701439", + "parent_user_id": "U05QNRSQW1E" + }, + { + "client_msg_id": "10c815a8-7d6c-45c6-9395-322d85226ab9", + "type": "message", + "text": "I think Marquez is running on WSL while you're trying to connect from host computer?", + "user": "U01RA9B5GG2", + "ts": "1693317758.935859", + "blocks": [ + { + "type": "rich_text", + "block_id": "L0TPP", + "elements": [ + { + "type": "rich_text_section", + "elements": [ + { + "type": "text", + "text": "I think Marquez is running on WSL while you're trying to connect from host computer?" + } + ] + } + ] + } + ], + "team": "T01CWUYP5AR", + "thread_ts": "1693293484.701439", + "parent_user_id": "U05QNRSQW1E" + } + ] + }, + { + "client_msg_id": "1a525866-90f7-4512-8bdb-c2e01d6ce8f5", + "type": "message", + "text": "New on the OpenLineage blog: , including:\n• the critical improvements it brings to the integration\n• the high-level design\n• implementation details\n• an example operator\n• planned enhancements\n• a list of supported operators\n• more.\nThe post, by <@U01RA9B5GG2>, <@U01DCLP0GU9> and myself is live now on the OpenLineage blog.", + "user": "U02LXF3HUN7", + "ts": "1693267537.810959", + "blocks": [ + { + "type": "rich_text", + "block_id": "VOPjI", + "elements": [ + { + "type": "rich_text_section", + "elements": [ + { + "type": "text", + "text": "New on the OpenLineage blog: " + }, + { + "type": "link", + "url": "https://openlineage.io/blog/airflow-provider", + "text": "a close look at the new OpenLineage Airflow Provider" + }, + { + "type": "text", + "text": ", including:\n" + } + ] + }, + { + "type": "rich_text_list", + "elements": [ + { + "type": "rich_text_section", + "elements": [ + { + "type": "text", + "text": " the critical improvements it brings to the integration" + } + ] + }, + { + "type": "rich_text_section", + "elements": [ + { + "type": "text", + "text": "the high-level design" + } + ] + }, + { + "type": "rich_text_section", + "elements": [ + { + "type": "text", + "text": "implementation details" + } + ] + }, + { + "type": "rich_text_section", + "elements": [ + { + "type": "text", + "text": "an example operator" + } + ] + }, + { + "type": "rich_text_section", + "elements": [ + { + "type": "text", + "text": "planned enhancements" + } + ] + }, + { + "type": "rich_text_section", + "elements": [ + { + "type": "text", + "text": "a list of supported operators" + } + ] + }, + { + "type": "rich_text_section", + "elements": [ + { + "type": "text", + "text": "more." + } + ] + } + ], + "style": "bullet", + "indent": 0, + "border": 0 + }, + { + "type": "rich_text_section", + "elements": [ + { + "type": "text", + "text": "The " + }, + { + "type": "text", + "text": "post", + "style": { + "unlink": true + } + }, + { + "type": "text", + "text": ", by " + }, + { + "type": "user", + "user_id": "U01RA9B5GG2" + }, + { + "type": "text", + "text": ", " + }, + { + "type": "user", + "user_id": "U01DCLP0GU9" + }, + { + "type": "text", + "text": " and myself is live now on the OpenLineage blog." + } + ] + } + ] + } + ], + "team": "T01CWUYP5AR", + "attachments": [ + { + "from_url": "https://openlineage.io/blog/airflow-provider", + "service_icon": "https://openlineage.io/img/favicon.ico", + "ts": 1692748800, + "id": 1, + "original_url": "https://openlineage.io/blog/airflow-provider", + "fallback": "The OpenLineage Airflow Provider is Here | OpenLineage", + "text": "Built-in OpenLineage support in Airflow means big improvements in reliability, lineage output, and custom operator implementation.", + "title": "The OpenLineage Airflow Provider is Here | OpenLineage", + "title_link": "https://openlineage.io/blog/airflow-provider", + "service_name": "openlineage.io" + } + ], + "reactions": [ + { + "name": "tada", + "users": [ + "U05P973CGHZ", + "U01HNKK4XAM", + "U01RA9B5GG2", + "U0544QC1DS9", + "U01HVNU6A4C" + ], + "count": 5 + } + ] + }, + { + "client_msg_id": "3efe9a2f-372c-41e9-8a46-f1c88615977f", + "type": "message", + "text": "\nThe agenda for the on 9/18 has been updated. This promises to be an exciting, richly productive discussion. Don’t miss it if you’ll be in the area!\n1. Intros\n2. Evolution of spec presentation/discussion (project background/history)\n3. State of the community\n4. Spark/Column lineage update\n5. Airflow Provider update \n6. Roadmap Discussion\n7. Action items review/next steps\n", + "user": "U02LXF3HUN7", + "ts": "1693258111.112809", + "blocks": [ + { + "type": "rich_text", + "block_id": "pEYP", + "elements": [ + { + "type": "rich_text_section", + "elements": [ + { + "type": "broadcast", + "range": "channel" + }, + { + "type": "text", + "text": "\nThe agenda for the " + }, + { + "type": "link", + "url": "https://www.meetup.com/openlineage/events/295488014/?utm_medium=referral&utm_campaign=share-btn_savedevents_share_modal&utm_source=link", + "text": "Toronto Meetup at Airflow Summit" + }, + { + "type": "text", + "text": " on 9/18 has been updated. This promises to be an exciting, richly productive discussion. Don’t miss it if you’ll be in the area!\n" + } + ] + }, + { + "type": "rich_text_list", + "elements": [ + { + "type": "rich_text_section", + "elements": [ + { + "type": "text", + "text": "Intros" + } + ] + }, + { + "type": "rich_text_section", + "elements": [ + { + "type": "text", + "text": "Evolution of spec presentation/discussion (project background/history)" + } + ] + }, + { + "type": "rich_text_section", + "elements": [ + { + "type": "text", + "text": "State of the community" + } + ] + }, + { + "type": "rich_text_section", + "elements": [ + { + "type": "text", + "text": "Spark/Column lineage update" + } + ] + }, + { + "type": "rich_text_section", + "elements": [ + { + "type": "text", + "text": "Airflow Provider update " + } + ] + }, + { + "type": "rich_text_section", + "elements": [ + { + "type": "text", + "text": "Roadmap Discussion" + } + ] + }, + { + "type": "rich_text_section", + "elements": [ + { + "type": "text", + "text": "Action items review/next steps" + } + ] + } + ], + "style": "ordered", + "indent": 0, + "border": 0 + }, + { + "type": "rich_text_section", + "elements": [] + } + ] + } + ], + "team": "T01CWUYP5AR", + "attachments": [ + { + "from_url": "https://www.meetup.com/openlineage/events/295488014/?utm_medium=referral&utm_campaign=share-btn_savedevents_share_modal&utm_source=link", + "image_url": "https://secure.meetupstatic.com/photos/event/5/4/2/d/600_515181549.jpeg", + "image_width": 600, + "image_height": 338, + "image_bytes": 16248, + "service_icon": "https://secure.meetupstatic.com/next/images/general/m_swarm_120x120.png", + "id": 1, + "original_url": "https://www.meetup.com/openlineage/events/295488014/?utm_medium=referral&utm_campaign=share-btn_savedevents_share_modal&utm_source=link", + "fallback": "Meetup: Toronto OpenLineage Meetup at Airflow Summit, Mon, Sep 18, 2023, 2:00 PM | Meetup", + "text": "Data engineers and pipeline managers know that producing data lineage – end-to-end pipeline metadata instrumented at runtime or parsed at design time – is a heavy lift with", + "title": "Toronto OpenLineage Meetup at Airflow Summit, Mon, Sep 18, 2023, 2:00 PM | Meetup", + "title_link": "https://www.meetup.com/openlineage/events/295488014/?utm_medium=referral&utm_campaign=share-btn_savedevents_share_modal&utm_source=link", + "service_name": "Meetup" + } + ], + "reactions": [ + { + "name": "heart", + "users": [ + "U0282HEEHB8", + "U02MK6YNAQ5", + "U055N2GRT4P" + ], + "count": 3 + } + ] + }, + { + "client_msg_id": "fddd2e84-0315-454b-949c-beed8a7eed48", + "type": "message", + "text": "Hi folks. I have some pure golang jobs from which I need to emit OL events to Marquez. Is the right way to go about this to generate a Golang client from the Marquez OpenAPI spec and use that client from my go jobs?", + "user": "U05PVS8GRJ6", + "ts": "1693243558.640159", + "blocks": [ + { + "type": "rich_text", + "block_id": "O4/r", + "elements": [ + { + "type": "rich_text_section", + "elements": [ + { + "type": "text", + "text": "Hi folks. I have some pure golang jobs from which I need to emit OL events to Marquez. Is the right way to go about this to generate a Golang client from the Marquez OpenAPI spec and use that client from my go jobs?" + } + ] + } + ] + } + ], + "team": "T01CWUYP5AR", + "thread_ts": "1693243558.640159", + "reply_count": 3, + "reply_users_count": 2, + "latest_reply": "1693251005.275449", + "reply_users": [ + "U02S6F54MAB", + "U05PVS8GRJ6" + ], + "is_locked": false, + "subscribed": false, + "replies": [ + { + "client_msg_id": "12fbc59c-cee7-48d7-96c8-42c4daa5ab3b", + "type": "message", + "text": "I'd rather generate them from OL spec (compliant with JSON Schema)", + "user": "U02S6F54MAB", + "ts": "1693247004.031009", + "blocks": [ + { + "type": "rich_text", + "block_id": "dKe", + "elements": [ + { + "type": "rich_text_section", + "elements": [ + { + "type": "text", + "text": "I'd rather generate them from OL spec (compliant with JSON Schema)" + } + ] + } + ] + } + ], + "team": "T01CWUYP5AR", + "thread_ts": "1693243558.640159", + "parent_user_id": "U05PVS8GRJ6" + }, + { + "client_msg_id": "e249d204-d3b7-4f44-94de-e9efb608a17f", + "type": "message", + "text": "I'll look into this. I take you to mean that I would use the OL spec which is available as a set of JSON schemas to create the data object and then HTTP POST it using vanilla Golang. Is that correct? Thank you for your help!", + "user": "U05PVS8GRJ6", + "ts": "1693249941.395039", + "blocks": [ + { + "type": "rich_text", + "block_id": "hlNf", + "elements": [ + { + "type": "rich_text_section", + "elements": [ + { + "type": "text", + "text": "I'll look into this. I take you to mean that I would use the OL spec which is available as a set of JSON schemas to create the data object and then HTTP POST it using vanilla Golang. Is that correct? Thank you for your help!" + } + ] + } + ] + } + ], + "team": "T01CWUYP5AR", + "edited": { + "user": "U05PVS8GRJ6", + "ts": "1693249952.000000" + }, + "thread_ts": "1693243558.640159", + "parent_user_id": "U05PVS8GRJ6" + }, + { + "client_msg_id": "a2580fde-0703-4859-aad9-72cf2f4e4008", + "type": "message", + "text": "Correct! You’re also very welcome to contribute Golang client (currently we have Python & Java clients) if you manage to send events using golang :slightly_smiling_face:", + "user": "U02S6F54MAB", + "ts": "1693251005.275449", + "blocks": [ + { + "type": "rich_text", + "block_id": "8ICeX", + "elements": [ + { + "type": "rich_text_section", + "elements": [ + { + "type": "text", + "text": "Correct! You’re also very welcome to contribute Golang client (currently we have Python & Java clients) if you manage to send events using golang " + }, + { + "type": "emoji", + "name": "slightly_smiling_face", + "unicode": "1f642" + } + ] + } + ] + } + ], + "team": "T01CWUYP5AR", + "thread_ts": "1693243558.640159", + "parent_user_id": "U05PVS8GRJ6", + "reactions": [ + { + "name": "clap", + "users": [ + "U05PVS8GRJ6" + ], + "count": 1 + } + ] + } + ] + }, + { + "client_msg_id": "55149130-15e5-4a9d-8832-af9e9ba9541c", + "type": "message", + "text": "and something else: I understand that Marquez does not yet support the 2.0 spec, hence it's incompatible with static metadata right? I tried to emit a list of `DatasetEvent` s and got `HTTPError: 422 Client Error: Unprocessable Entity for url: ` (I'm using a `FileTransport` for now)", + "user": "U05HFGKEYVB", + "ts": "1693212561.369509", + "blocks": [ + { + "type": "rich_text", + "block_id": "SZo", + "elements": [ + { + "type": "rich_text_section", + "elements": [ + { + "type": "text", + "text": "and something else: I understand that Marquez does not yet support the 2.0 spec, hence it's incompatible with static metadata right? I tried to emit a list of " + }, + { + "type": "text", + "text": "DatasetEvent", + "style": { + "code": true + } + }, + { + "type": "text", + "text": " s and got " + }, + { + "type": "text", + "text": "HTTPError: 422 Client Error: Unprocessable Entity for url: ", + "style": { + "code": true + } + }, + { + "type": "link", + "url": "http://localhost:3000/api/v1/lineage", + "style": { + "code": true + } + }, + { + "type": "text", + "text": " (I'm using a " + }, + { + "type": "text", + "text": "FileTransport", + "style": { + "code": true + } + }, + { + "type": "text", + "text": " for now)" + } + ] + } + ] + } + ], + "team": "T01CWUYP5AR", + "edited": { + "user": "U05HFGKEYVB", + "ts": "1693212597.000000" + }, + "thread_ts": "1693212561.369509", + "reply_count": 3, + "reply_users_count": 2, + "latest_reply": "1693217141.924759", + "reply_users": [ + "U02MK6YNAQ5", + "U05HFGKEYVB" + ], + "is_locked": false, + "subscribed": false, + "replies": [ + { + "client_msg_id": "61807027-58e2-425f-a65e-9af09cc49346", + "type": "message", + "text": "marquez is not capable of reflecting `DatasetEvents` in DB but it should respond with `Unsupported event type`", + "user": "U02MK6YNAQ5", + "ts": "1693216969.231179", + "blocks": [ + { + "type": "rich_text", + "block_id": "10Wp8", + "elements": [ + { + "type": "rich_text_section", + "elements": [ + { + "type": "text", + "text": "marquez is not capable of reflecting " + }, + { + "type": "text", + "text": "DatasetEvents", + "style": { + "code": true + } + }, + { + "type": "text", + "text": " in DB but it should respond with " + }, + { + "type": "text", + "text": "Unsupported event type", + "style": { + "code": true + } + } + ] + } + ] + } + ], + "team": "T01CWUYP5AR", + "thread_ts": "1693212561.369509", + "parent_user_id": "U05HFGKEYVB" + }, + { + "client_msg_id": "9ce1c23c-1210-4d7a-85b6-66a829fb4d6f", + "type": "message", + "text": "and return 200 instead of `201` created", + "user": "U02MK6YNAQ5", + "ts": "1693216995.122589", + "blocks": [ + { + "type": "rich_text", + "block_id": "wcHt9", + "elements": [ + { + "type": "rich_text_section", + "elements": [ + { + "type": "text", + "text": "and return 200 instead of " + }, + { + "type": "text", + "text": "201", + "style": { + "code": true + } + }, + { + "type": "text", + "text": " created" + } + ] + } + ] + } + ], + "team": "T01CWUYP5AR", + "thread_ts": "1693212561.369509", + "parent_user_id": "U05HFGKEYVB" + }, + { + "client_msg_id": "c054a5c8-3e0f-4de9-b92f-e769fb9e024a", + "type": "message", + "text": "I'll have a deeper look then, probably I'm doing something wrong. thanks <@U02MK6YNAQ5>", + "user": "U05HFGKEYVB", + "ts": "1693217141.924759", + "blocks": [ + { + "type": "rich_text", + "block_id": "eKH7I", + "elements": [ + { + "type": "rich_text_section", + "elements": [ + { + "type": "text", + "text": "I'll have a deeper look then, probably I'm doing something wrong. thanks " + }, + { + "type": "user", + "user_id": "U02MK6YNAQ5" + } + ] + } + ] + } + ], + "team": "T01CWUYP5AR", + "thread_ts": "1693212561.369509", + "parent_user_id": "U05HFGKEYVB" + } + ] + }, + { + "client_msg_id": "e403e4d5-e318-4c72-bba4-aa1fbebe40a8", + "type": "message", + "text": "hi folks, I'm looking into exporting static metadata, and found that `DatasetEvent` requires a `eventTime`, which in my mind doesn't make sense for static events. I'm setting it to `None` and the Python client seems to work, but wanted to ask if I'm missing something.", + "user": "U05HFGKEYVB", + "ts": "1693212473.810659", + "blocks": [ + { + "type": "rich_text", + "block_id": "EBPk", + "elements": [ + { + "type": "rich_text_section", + "elements": [ + { + "type": "text", + "text": "hi folks, I'm looking into exporting static metadata, and found that " + }, + { + "type": "text", + "text": "DatasetEvent", + "style": { + "code": true + } + }, + { + "type": "text", + "text": " requires a " + }, + { + "type": "text", + "text": "eventTime", + "style": { + "code": true + } + }, + { + "type": "text", + "text": ", which in my mind doesn't make sense for static events. I'm setting it to " + }, + { + "type": "text", + "text": "None", + "style": { + "code": true + } + }, + { + "type": "text", + "text": " and the Python client seems to work, but wanted to ask if I'm missing something." + } + ] + } + ] + } + ], + "team": "T01CWUYP5AR", + "thread_ts": "1693212473.810659", + "reply_count": 3, + "reply_users_count": 2, + "latest_reply": "1693216913.315869", + "reply_users": [ + "U02MK6YNAQ5", + "U05HFGKEYVB" + ], + "is_locked": false, + "subscribed": false, + "replies": [ + { + "client_msg_id": "c7d4fc24-9e83-44ee-b43a-c1befb6434e2", + "type": "message", + "text": "Although you emit `DatasetEvent`, you still emit an event and `eventTime` is a valid marker.", + "user": "U02MK6YNAQ5", + "ts": "1693216750.036829", + "blocks": [ + { + "type": "rich_text", + "block_id": "r2Ree", + "elements": [ + { + "type": "rich_text_section", + "elements": [ + { + "type": "text", + "text": "Although you emit " + }, + { + "type": "text", + "text": "DatasetEvent", + "style": { + "code": true + } + }, + { + "type": "text", + "text": ", you still emit an event and " + }, + { + "type": "text", + "text": "eventTime", + "style": { + "code": true + } + }, + { + "type": "text", + "text": " is a valid marker." + } + ] + } + ] + } + ], + "team": "T01CWUYP5AR", + "thread_ts": "1693212473.810659", + "parent_user_id": "U05HFGKEYVB" + }, + { + "client_msg_id": "d6389b59-61f0-4cb1-91fc-50a90c9d1d6c", + "type": "message", + "text": "so, should I use the current time at the moment of emitting it and that's it?", + "user": "U05HFGKEYVB", + "ts": "1693216900.409389", + "blocks": [ + { + "type": "rich_text", + "block_id": "Gr6M", + "elements": [ + { + "type": "rich_text_section", + "elements": [ + { + "type": "text", + "text": "so, should I use the current time at the moment of emitting it and that's it?" + } + ] + } + ] + } + ], + "team": "T01CWUYP5AR", + "thread_ts": "1693212473.810659", + "parent_user_id": "U05HFGKEYVB" + }, + { + "client_msg_id": "d22e44aa-b4b7-43af-bcd9-11183e637c33", + "type": "message", + "text": "yes, that should be it", + "user": "U02MK6YNAQ5", + "ts": "1693216913.315869", + "blocks": [ + { + "type": "rich_text", + "block_id": "I0McS", + "elements": [ + { + "type": "rich_text_section", + "elements": [ + { + "type": "text", + "text": "yes, that should be it" + } + ] + } + ] + } + ], + "team": "T01CWUYP5AR", + "thread_ts": "1693212473.810659", + "parent_user_id": "U05HFGKEYVB", + "reactions": [ + { + "name": "gratitude-thank-you", + "users": [ + "U05HFGKEYVB" + ], + "count": 1 + } + ] + } + ] + }, + { + "client_msg_id": "5a0c0a38-e883-47de-bb00-09784e9edc84", + "type": "message", + "text": "hi Openlineage team , we would like to join one of your meetups(me and <@U05HK41VCH1> nad @Phil Rolph and we're wondering if you are hosting any meetups after the 18/9 ? We are trying to join this but air - tickets are quite expensive", + "user": "U05J5GRKY10", + "ts": "1692975450.380969", + "blocks": [ + { + "type": "rich_text", + "block_id": "vhZR", + "elements": [ + { + "type": "rich_text_section", + "elements": [ + { + "type": "text", + "text": "hi Openlineage team , we would like to join one of your meetups(me and " + }, + { + "type": "user", + "user_id": "U05HK41VCH1" + }, + { + "type": "text", + "text": " nad @Phil Rolph and we're wondering if you are hosting any meetups after the 18/9 ? We are trying to join this but air - tickets are quite expensive" + } + ] + } + ] + } + ], + "team": "T01CWUYP5AR", + "thread_ts": "1692975450.380969", + "reply_count": 4, + "reply_users_count": 3, + "latest_reply": "1692978699.479329", + "reply_users": [ + "U01HNKK4XAM", + "U05J5GRKY10", + "U02LXF3HUN7" + ], + "is_locked": false, + "subscribed": true, + "last_read": "1692978699.479329", + "replies": [ + { + "client_msg_id": "d90db7c4-ea60-4eff-a0b3-24568c51f105", + "type": "message", + "text": "there will certainly be more meetups, don’t worry about that!", + "user": "U01HNKK4XAM", + "ts": "1692977532.949389", + "blocks": [ + { + "type": "rich_text", + "block_id": "RF3H", + "elements": [ + { + "type": "rich_text_section", + "elements": [ + { + "type": "text", + "text": "there will certainly be more meetups, don’t worry about that!" + } + ] + } + ] + } + ], + "team": "T01CWUYP5AR", + "thread_ts": "1692975450.380969", + "parent_user_id": "U05J5GRKY10" + }, + { + "client_msg_id": "47c313f7-9d2c-4b8f-9b55-7e20ee01debc", + "type": "message", + "text": "where are you located? perhaps we can try to organize a meetup closer to where you are.", + "user": "U01HNKK4XAM", + "ts": "1692977550.649789", + "blocks": [ + { + "type": "rich_text", + "block_id": "Um2m", + "elements": [ + { + "type": "rich_text_section", + "elements": [ + { + "type": "text", + "text": "where are you located? perhaps we can try to organize a meetup closer to where you are." + } + ] + } + ] + } + ], + "team": "T01CWUYP5AR", + "thread_ts": "1692975450.380969", + "parent_user_id": "U05J5GRKY10" + }, + { + "client_msg_id": "ef556e07-2f4c-432f-a51a-a129ea8e7274", + "type": "message", + "text": "Thanks a lot for the response, we are in London. We'd be glad to help you organise a meetup and also meet in person!", + "user": "U05J5GRKY10", + "ts": "1692978577.530149", + "blocks": [ + { + "type": "rich_text", + "block_id": "UEDw", + "elements": [ + { + "type": "rich_text_section", + "elements": [ + { + "type": "text", + "text": "Thanks a lot for the response, we are in London. We'd be glad to help you organise a meetup and also meet in person!" + } + ] + } + ] + } + ], + "team": "T01CWUYP5AR", + "thread_ts": "1692975450.380969", + "parent_user_id": "U05J5GRKY10" + }, + { + "client_msg_id": "a8f02080-dd7e-4a05-a624-02656ebea128", + "type": "message", + "text": "This is awesome, thanks <@U05J5GRKY10>. I’ll start a channel and invite you", + "user": "U02LXF3HUN7", + "ts": "1692978699.479329", + "blocks": [ + { + "type": "rich_text", + "block_id": "MiX", + "elements": [ + { + "type": "rich_text_section", + "elements": [ + { + "type": "text", + "text": "This is awesome, thanks " + }, + { + "type": "user", + "user_id": "U05J5GRKY10" + }, + { + "type": "text", + "text": ". I’ll start a channel and invite you" + } + ] + } + ] + } + ], + "team": "T01CWUYP5AR", + "thread_ts": "1692975450.380969", + "parent_user_id": "U05J5GRKY10" + } + ] + }, + { + "client_msg_id": "597a7abe-590e-4827-accc-276859f36a52", + "type": "message", + "text": "\nFriendly reminder: our next in-person meetup is next Wednesday, August 30th in San Francisco at Astronomer’s offices in the Financial District. You can sign up and find the details on the .", + "user": "U02LXF3HUN7", + "ts": "1692973763.570629", + "blocks": [ + { + "type": "rich_text", + "block_id": "T3fR", + "elements": [ + { + "type": "rich_text_section", + "elements": [ + { + "type": "broadcast", + "range": "channel" + }, + { + "type": "text", + "text": "\nFriendly reminder: our next in-person meetup is next Wednesday, August 30th in San Francisco at Astronomer’s offices in the Financial District. You can sign up and find the details on the " + }, + { + "type": "link", + "url": "https://www.meetup.com/meetup-group-bnfqymxe/events/295195280/?utm_medium=referral&utm_campaign=share-btn_savedevents_share_modal&utm_source=link", + "text": "meetup event page" + }, + { + "type": "text", + "text": "." + } + ] + } + ] + } + ], + "team": "T01CWUYP5AR", + "attachments": [ + { + "from_url": "https://www.meetup.com/meetup-group-bnfqymxe/events/295195280/?utm_medium=referral&utm_campaign=share-btn_savedevents_share_modal&utm_source=link", + "image_url": "https://secure.meetupstatic.com/photos/event/b/d/0/c/600_514848396.jpeg", + "image_width": 600, + "image_height": 338, + "image_bytes": 13814, + "service_icon": "https://secure.meetupstatic.com/next/images/general/m_swarm_120x120.png", + "id": 1, + "original_url": "https://www.meetup.com/meetup-group-bnfqymxe/events/295195280/?utm_medium=referral&utm_campaign=share-btn_savedevents_share_modal&utm_source=link", + "fallback": "Meetup: OpenLineage Meetup @ Astronomer, Wed, Aug 30, 2023, 5:30 PM | Meetup", + "text": "Data engineers and pipeline managers know that producing data lineage – end-to-end pipeline metadata instrumented at runtime or parsed at design time – is a heavy lift with", + "title": "OpenLineage Meetup @ Astronomer, Wed, Aug 30, 2023, 5:30 PM | Meetup", + "title_link": "https://www.meetup.com/meetup-group-bnfqymxe/events/295195280/?utm_medium=referral&utm_campaign=share-btn_savedevents_share_modal&utm_source=link", + "service_name": "Meetup" + } + ] + }, + { + "client_msg_id": "e6f83c78-bda8-445a-ac52-5511e8e9e1e1", + "type": "message", + "text": "\nWe released OpenLineage 1.1.0, including:\nAdditions:\n• Flink: create Openlineage configuration based on Flink configuration `#2033` \n• Java: add Javadocs to the Java client `#2004` \n• Spark: append output dataset name to a job name `#2036` \n• Spark: support Spark 3.4.1 `#2057` \nFixes:\n• Flink: fix a bug when getting schema for `KafkaSink` `#2042` \n• Spark: fix ignored event `adaptive_spark_plan` in Databricks `#2061` \nPlus additional bug fixes, doc changes and more.\nThanks to all the contributors, especially new contributors @pentium3 and <@U05HBLE7YPL>!\n*Release:* \n*Changelog:* \n*Commit history:* \n*Maven:* \n*PyPI:* ", + "user": "U02LXF3HUN7", + "ts": "1692817450.338859", + "blocks": [ + { + "type": "rich_text", + "block_id": "zaQ", + "elements": [ + { + "type": "rich_text_section", + "elements": [ + { + "type": "broadcast", + "range": "channel" + }, + { + "type": "text", + "text": "\nWe released OpenLineage 1.1.0, including:\nAdditions:\n" + } + ] + }, + { + "type": "rich_text_list", + "elements": [ + { + "type": "rich_text_section", + "elements": [ + { + "type": "text", + "text": "Flink: create Openlineage configuration based on Flink configuration " + }, + { + "type": "text", + "text": "#2033", + "style": { + "code": true + } + }, + { + "type": "text", + "text": " " + }, + { + "type": "link", + "url": "https://github.com/pawel-big-lebowski", + "text": "@pawel-big-lebowski", + "unsafe": true + } + ] + }, + { + "type": "rich_text_section", + "elements": [ + { + "type": "text", + "text": "Java: add Javadocs to the Java client " + }, + { + "type": "text", + "text": "#2004", + "style": { + "code": true + } + }, + { + "type": "text", + "text": " " + }, + { + "type": "link", + "url": "https://github.com/julienledem", + "text": "@julienledem", + "unsafe": true + } + ] + }, + { + "type": "rich_text_section", + "elements": [ + { + "type": "text", + "text": "Spark: append output dataset name to a job name " + }, + { + "type": "text", + "text": "#2036", + "style": { + "code": true + } + }, + { + "type": "text", + "text": " " + }, + { + "type": "link", + "url": "https://github.com/pawel-big-lebowski", + "text": "@pawel-big-lebowski", + "unsafe": true + } + ] + }, + { + "type": "rich_text_section", + "elements": [ + { + "type": "text", + "text": "Spark: support Spark 3.4.1 " + }, + { + "type": "text", + "text": "#2057", + "style": { + "code": true + } + }, + { + "type": "text", + "text": " " + }, + { + "type": "link", + "url": "https://github.com/pawel-big-lebowski", + "text": "@pawel-big-lebowski", + "unsafe": true + } + ] + } + ], + "style": "bullet", + "indent": 0, + "border": 0 + }, + { + "type": "rich_text_section", + "elements": [ + { + "type": "text", + "text": "Fixes:\n" + } + ] + }, + { + "type": "rich_text_list", + "elements": [ + { + "type": "rich_text_section", + "elements": [ + { + "type": "text", + "text": "Flink: fix a bug when getting schema for " + }, + { + "type": "text", + "text": "KafkaSink", + "style": { + "code": true + } + }, + { + "type": "text", + "text": " " + }, + { + "type": "text", + "text": "#2042", + "style": { + "code": true + } + }, + { + "type": "text", + "text": " " + }, + { + "type": "link", + "url": "https://github.com/pentium3", + "text": "@pentium3", + "unsafe": true + } + ] + }, + { + "type": "rich_text_section", + "elements": [ + { + "type": "text", + "text": "Spark: fix ignored event " + }, + { + "type": "text", + "text": "adaptive_spark_plan", + "style": { + "code": true + } + }, + { + "type": "text", + "text": " in Databricks " + }, + { + "type": "text", + "text": "#2061", + "style": { + "code": true + } + }, + { + "type": "text", + "text": " " + }, + { + "type": "link", + "url": "https://github.com/algorithmy1", + "text": "@algorithmy1", + "unsafe": true + } + ] + } + ], + "style": "bullet", + "indent": 0, + "border": 0 + }, + { + "type": "rich_text_section", + "elements": [ + { + "type": "text", + "text": "Plus additional bug fixes, doc changes and more.\nThanks to all the contributors, especially new contributors @pentium3 and " + }, + { + "type": "user", + "user_id": "U05HBLE7YPL" + }, + { + "type": "text", + "text": "!\n" + }, + { + "type": "text", + "text": "Release:", + "style": { + "bold": true + } + }, + { + "type": "text", + "text": " " + }, + { + "type": "link", + "url": "https://github.com/OpenLineage/OpenLineage/releases/tag/1.1.0" + }, + { + "type": "text", + "text": "\n" + }, + { + "type": "text", + "text": "Changelog: ", + "style": { + "bold": true + } + }, + { + "type": "link", + "url": "https://github.com/OpenLineage/OpenLineage/blob/main/CHANGELOG.md" + }, + { + "type": "text", + "text": "\n" + }, + { + "type": "text", + "text": "Commit history:", + "style": { + "bold": true + } + }, + { + "type": "text", + "text": " " + }, + { + "type": "link", + "url": "https://github.com/OpenLineage/OpenLineage/compare/1.0.0...1.1.0" + }, + { + "type": "text", + "text": "\n" + }, + { + "type": "text", + "text": "Maven:", + "style": { + "bold": true + } + }, + { + "type": "text", + "text": " " + }, + { + "type": "link", + "url": "https://oss.sonatype.org/#nexus-search;quick~openlineage" + }, + { + "type": "text", + "text": "\n" + }, + { + "type": "text", + "text": "PyPI:", + "style": { + "bold": true + } + }, + { + "type": "text", + "text": " " + }, + { + "type": "link", + "url": "https://pypi.org/project/openlineage-python/" + } + ] + } + ] + } + ], + "team": "T01CWUYP5AR", + "reactions": [ + { + "name": "clap", + "users": [ + "U05P34LH669", + "U05HBLE7YPL", + "U058ZJBGSAJ", + "U05JBHLPY8K", + "U01HVNU6A4C", + "U01RA9B5GG2", + "U01HNKK4XAM", + "U05EN2CKBS8", + "U05G1LTK0F2" + ], + "count": 9 + }, + { + "name": "gratitude-thank-you", + "users": [ + "U05JY6MN8MS" + ], + "count": 1 + } + ] + }, + { + "client_msg_id": "571ffe0f-9038-4345-8581-052ad7c5199e", + "type": "message", + "text": "Hey folks! Do we have clear step-by-step documentation on how we can leverage the `ServiceLoader` based approach for injecting specific OpenLineage customisations for tweaking the transport type with defaults / tweaking column level lineage etc?", + "user": "U05JBHLPY8K", + "ts": "1692810528.463669", + "blocks": [ + { + "type": "rich_text", + "block_id": "ESmZ", + "elements": [ + { + "type": "rich_text_section", + "elements": [ + { + "type": "text", + "text": "Hey folks! Do we have clear step-by-step documentation on how we can leverage the " + }, + { + "type": "text", + "text": "ServiceLoader", + "style": { + "code": true + } + }, + { + "type": "text", + "text": " based approach for injecting specific OpenLineage customisations for tweaking the transport type with defaults / tweaking column level lineage etc?" + } + ] + } + ] + } + ], + "team": "T01CWUYP5AR", + "thread_ts": "1692810528.463669", + "reply_count": 7, + "reply_users_count": 2, + "latest_reply": "1692960750.563679", + "reply_users": [ + "U01RA9B5GG2", + "U05JBHLPY8K" + ], + "is_locked": false, + "subscribed": false, + "replies": [ + { + "client_msg_id": "c069a4de-86a9-43d2-84dd-df5ef9129a82", + "type": "message", + "text": "This proposal - ", + "user": "U01RA9B5GG2", + "ts": "1692811472.232389", + "blocks": [ + { + "type": "rich_text", + "block_id": "Gqz0", + "elements": [ + { + "type": "rich_text_section", + "elements": [ + { + "type": "text", + "text": "This proposal - " + }, + { + "type": "link", + "url": "https://github.com/OpenLineage/OpenLineage/blob/41dbbf1799595bd9cd1567df0a7027de08619741/proposals/168/making_spark_visitors_extensible.md" + } + ] + } + ] + } + ], + "team": "T01CWUYP5AR", + "thread_ts": "1692810528.463669", + "parent_user_id": "U05JBHLPY8K" + }, + { + "client_msg_id": "ca29d7f7-b55b-4a88-bcd8-a80a1164af88", + "type": "message", + "text": "For custom transport, you have to provide implementation of interface and point to it in `META_INF` file", + "user": "U01RA9B5GG2", + "ts": "1692811745.594439", + "blocks": [ + { + "type": "rich_text", + "block_id": "4gw", + "elements": [ + { + "type": "rich_text_section", + "elements": [ + { + "type": "text", + "text": "For custom transport, you have to provide implementation of interface " + }, + { + "type": "link", + "url": "https://github.com/OpenLineage/OpenLineage/blob/4a1a5c3bf9767467b71ca0e1b6d820ba9e0a1a1d/client/java/src/main/java/io/openlineage/client/transports/TransportBuilder.java#L8", + "text": "https://github.com/OpenLineage/OpenLineage/blob/4a1a5c3bf9767467b71ca0e1b6d820ba9e[…]ain/java/io/openlineage/client/transports/TransportBuilder.java" + }, + { + "type": "text", + "text": " and point to it in " + }, + { + "type": "text", + "text": "META_INF", + "style": { + "code": true + } + }, + { + "type": "text", + "text": " file" + } + ] + } + ] + } + ], + "team": "T01CWUYP5AR", + "attachments": [ + { + "id": 1, + "footer_icon": "https://slack.github.com/static/img/favicon-neutral.png", + "color": "24292f", + "bot_id": "B01VA0FB340", + "app_unfurl_url": "https://github.com/OpenLineage/OpenLineage/blob/4a1a5c3bf9767467b71ca0e1b6d820ba9e0a1a1d/client/java/src/main/java/io/openlineage/client/transports/TransportBuilder.java#L8", + "is_app_unfurl": true, + "app_id": "A01BP7R4KNY", + "fallback": "", + "text": "```\npublic interface TransportBuilder {\n```", + "title": "", + "footer": "", + "mrkdwn_in": [ + "text" + ] + } + ], + "thread_ts": "1692810528.463669", + "parent_user_id": "U05JBHLPY8K" + }, + { + "client_msg_id": "73254d9f-6e38-47e0-884c-1ecef8c79fbb", + "type": "message", + "text": "But if I understand correctly, if you want to change behavior rather than extend, the correct way may be to either contribute it to repo - if that behavior is useful to anyone, or fork the repo", + "user": "U01RA9B5GG2", + "ts": "1692811792.937659", + "blocks": [ + { + "type": "rich_text", + "block_id": "8He", + "elements": [ + { + "type": "rich_text_section", + "elements": [ + { + "type": "text", + "text": "But if I understand correctly, if you want to change behavior rather than extend, the correct way may be to either contribute it to repo - if that behavior is useful to anyone, or fork the repo" + } + ] + } + ] + } + ], + "team": "T01CWUYP5AR", + "thread_ts": "1692810528.463669", + "parent_user_id": "U05JBHLPY8K" + }, + { + "client_msg_id": "117FDAFB-AA4A-489E-BAF1-FFE5AF2D0597", + "type": "message", + "text": "<@U01RA9B5GG2> - Can you elaborate more on the \"point to it in META_INF file\"? Let's say we have the custom transport type built in a standalone jar by extending transport builder - what're the exact next steps to use this custom transport in the standalone jar when doing spark-submit?", + "user": "U05JBHLPY8K", + "ts": "1692818083.000699", + "blocks": [ + { + "type": "rich_text", + "block_id": "Ani", + "elements": [ + { + "type": "rich_text_section", + "elements": [ + { + "type": "user", + "user_id": "U01RA9B5GG2" + }, + { + "type": "text", + "text": " - Can you elaborate more on the \"point to it in META_INF file\"? Let's say we have the custom transport type built in a standalone jar by extending transport builder - what're the exact next steps to use this custom transport in the standalone jar when doing spark-submit?" + } + ] + } + ] + } + ], + "team": "T01CWUYP5AR", + "thread_ts": "1692810528.463669", + "parent_user_id": "U05JBHLPY8K" + }, + { + "client_msg_id": "d79e461b-28bd-4d3b-b282-f400256e558b", + "type": "message", + "text": "<@U05JBHLPY8K> your jar needs to have `META-INF/services/io.openlineage.client.transports.TransportBuilder` with fully qualified class names of your custom TransportBuilders there - like `openlineage-spark` has\n```io.openlineage.client.transports.HttpTransportBuilder\nio.openlineage.client.transports.KafkaTransportBuilder\nio.openlineage.client.transports.ConsoleTransportBuilder\nio.openlineage.client.transports.FileTransportBuilder\nio.openlineage.client.transports.KinesisTransportBuilder```", + "user": "U01RA9B5GG2", + "ts": "1692818593.597789", + "blocks": [ + { + "type": "rich_text", + "block_id": "j7yIr", + "elements": [ + { + "type": "rich_text_section", + "elements": [ + { + "type": "user", + "user_id": "U05JBHLPY8K" + }, + { + "type": "text", + "text": " your jar needs to have " + }, + { + "type": "text", + "text": "META-INF/services/io.openlineage.client.transports.TransportBuilder", + "style": { + "code": true + } + }, + { + "type": "text", + "text": " with fully qualified class names of your custom TransportBuilders there - like `openlineage-spark` has\n" + } + ] + }, + { + "type": "rich_text_preformatted", + "elements": [ + { + "type": "text", + "text": "io.openlineage.client.transports.HttpTransportBuilder\nio.openlineage.client.transports.KafkaTransportBuilder\nio.openlineage.client.transports.ConsoleTransportBuilder\nio.openlineage.client.transports.FileTransportBuilder\nio.openlineage.client.transports.KinesisTransportBuilder" + } + ], + "border": 0 + } + ] + } + ], + "team": "T01CWUYP5AR", + "thread_ts": "1692810528.463669", + "parent_user_id": "U05JBHLPY8K" + }, + { + "client_msg_id": "101b3bd4-b008-4ea4-8731-fdf91e2472cf", + "type": "message", + "text": "<@U01RA9B5GG2> - I think this change may be required for consumers to leverage custom transports, can you check & verify this GH comment?\n", + "user": "U05JBHLPY8K", + "ts": "1692942569.835759", + "blocks": [ + { + "type": "rich_text", + "block_id": "SDPrA", + "elements": [ + { + "type": "rich_text_section", + "elements": [ + { + "type": "user", + "user_id": "U01RA9B5GG2" + }, + { + "type": "text", + "text": " - I think this change may be required for consumers to leverage custom transports, can you check & verify this GH comment?\n" + }, + { + "type": "link", + "url": "https://github.com/OpenLineage/OpenLineage/issues/2007#issuecomment-1690350630" + } + ] + } + ] + } + ], + "team": "T01CWUYP5AR", + "attachments": [ + { + "id": 1, + "footer_icon": "https://slack.github.com/static/img/favicon-neutral.png", + "ts": 1692811356, + "color": "24292f", + "bot_id": "B01VA0FB340", + "app_unfurl_url": "https://github.com/OpenLineage/OpenLineage/issues/2007#issuecomment-1690350630", + "is_app_unfurl": true, + "app_id": "A01BP7R4KNY", + "fallback": "Comment on #2007 [PROPOSAL] Ability to support custom injectable dynamic header generation class/logic for HTTP transports", + "text": " - I think the `Type` enum in the Transport.java class has default visibility, which makes it invisible for OpenLineage consumers who are trying to define a custom transport logic\n\n\n\nI think it'll be great if we can make this `Type` enum have public visibility. Also, can we add a `CUSTOM` Type here for folks who are trying to define a custom transport of their own?\n\nPlease lmk If the changes proposed seem to be fine, I can draft a PR for the same", + "title": "Comment on #2007 [PROPOSAL] Ability to support custom injectable dynamic header generation class/logic for HTTP transports", + "title_link": "https://github.com/OpenLineage/OpenLineage/issues/2007#issuecomment-1690350630", + "footer": "", + "mrkdwn_in": [ + "text" + ] + } + ], + "thread_ts": "1692810528.463669", + "parent_user_id": "U05JBHLPY8K" + }, + { + "client_msg_id": "76108108-f51e-4d0f-9f2a-4eaa1513436c", + "type": "message", + "text": "Probably, I will look at more details next week <@U05JBHLPY8K> as I'm in transit", + "user": "U01RA9B5GG2", + "ts": "1692960750.563679", + "blocks": [ + { + "type": "rich_text", + "block_id": "8a=1", + "elements": [ + { + "type": "rich_text_section", + "elements": [ + { + "type": "text", + "text": "Probably, I will look at more details next week " + }, + { + "type": "user", + "user_id": "U05JBHLPY8K" + }, + { + "type": "text", + "text": " as I'm in transit" + } + ] + } + ] + } + ], + "team": "T01CWUYP5AR", + "thread_ts": "1692810528.463669", + "parent_user_id": "U05JBHLPY8K", + "reactions": [ + { + "name": "+1", + "users": [ + "U05JBHLPY8K" + ], + "count": 1 + } + ] + } + ] + }, + { + "client_msg_id": "f35a05f2-10a6-4394-b1be-dc89175cbd68", + "type": "message", + "text": "Approve a new release please :slightly_smiling_face:\n• Fix spark integration filtering Databricks events. ", + "user": "U05HBLE7YPL", + "ts": "1692802510.386629", + "blocks": [ + { + "type": "rich_text", + "block_id": "PrcFP", + "elements": [ + { + "type": "rich_text_section", + "elements": [ + { + "type": "text", + "text": "Approve a new release please " + }, + { + "type": "emoji", + "name": "slightly_smiling_face", + "unicode": "1f642" + }, + { + "type": "text", + "text": "\n" + } + ] + }, + { + "type": "rich_text_list", + "elements": [ + { + "type": "rich_text_section", + "elements": [ + { + "type": "text", + "text": "Fix spark integration filtering Databricks events. " + } + ] + } + ], + "style": "bullet", + "indent": 0, + "border": 0 + } + ] + } + ], + "team": "T01CWUYP5AR", + "edited": { + "user": "U05HBLE7YPL", + "ts": "1692802621.000000" + }, + "thread_ts": "1692802510.386629", + "reply_count": 2, + "reply_users_count": 1, + "latest_reply": "1692810798.179659", + "reply_users": [ + "U02LXF3HUN7" + ], + "is_locked": false, + "subscribed": true, + "last_read": "1692810798.179659", + "reactions": [ + { + "name": "heavy_plus_sign", + "users": [ + "U05HBLE7YPL", + "U053LCT71BQ", + "U05MUN2MH2S", + "U05P34LH669", + "U05P34PFH7F", + "U02S6F54MAB", + "U02LXF3HUN7", + "U01HNKK4XAM", + "U01DCMDFHBK", + "U01RA9B5GG2", + "U01DCLP0GU9" + ], + "count": 11 + } + ], + "replies": [ + { + "client_msg_id": "334a3373-b7b4-4688-885c-557f86b0bf65", + "type": "message", + "text": "Thank you for requesting a release <@U05HBLE7YPL>. Three +1s from committers will authorize.", + "user": "U02LXF3HUN7", + "ts": "1692808035.154579", + "blocks": [ + { + "type": "rich_text", + "block_id": "q9Adm", + "elements": [ + { + "type": "rich_text_section", + "elements": [ + { + "type": "text", + "text": "Thank you for requesting a release " + }, + { + "type": "user", + "user_id": "U05HBLE7YPL" + }, + { + "type": "text", + "text": ". Three +1s from committers will authorize." + } + ] + } + ] + } + ], + "team": "T01CWUYP5AR", + "thread_ts": "1692802510.386629", + "parent_user_id": "U05HBLE7YPL", + "reactions": [ + { + "name": "raised_hands", + "users": [ + "U05HBLE7YPL" + ], + "count": 1 + } + ] + }, + { + "client_msg_id": "45804461-4a1f-4c8e-8cdb-eca19c75336a", + "type": "message", + "text": "Thanks, all. The release is authorized and will be initiated within 2 business days.", + "user": "U02LXF3HUN7", + "ts": "1692810798.179659", + "blocks": [ + { + "type": "rich_text", + "block_id": "+5Iq", + "elements": [ + { + "type": "rich_text_section", + "elements": [ + { + "type": "text", + "text": "Thanks, all. The release is authorized and will be initiated within 2 business days." + } + ] + } + ] + } + ], + "team": "T01CWUYP5AR", + "thread_ts": "1692802510.386629", + "parent_user_id": "U05HBLE7YPL" + } + ] + } +] \ No newline at end of file diff --git a/slack-archive/data/C01NAFMBVEY.json b/slack-archive/data/C01NAFMBVEY.json new file mode 100644 index 0000000..0637a08 --- /dev/null +++ b/slack-archive/data/C01NAFMBVEY.json @@ -0,0 +1 @@ +[] \ No newline at end of file diff --git a/slack-archive/data/C030F1J0264.json b/slack-archive/data/C030F1J0264.json new file mode 100644 index 0000000..0637a08 --- /dev/null +++ b/slack-archive/data/C030F1J0264.json @@ -0,0 +1 @@ +[] \ No newline at end of file diff --git a/slack-archive/data/C04E3Q18RR9.json b/slack-archive/data/C04E3Q18RR9.json new file mode 100644 index 0000000..0637a08 --- /dev/null +++ b/slack-archive/data/C04E3Q18RR9.json @@ -0,0 +1 @@ +[] \ No newline at end of file diff --git a/slack-archive/data/C04JPTTC876.json b/slack-archive/data/C04JPTTC876.json new file mode 100644 index 0000000..0637a08 --- /dev/null +++ b/slack-archive/data/C04JPTTC876.json @@ -0,0 +1 @@ +[] \ No newline at end of file diff --git a/slack-archive/data/C04QSV0GG23.json b/slack-archive/data/C04QSV0GG23.json new file mode 100644 index 0000000..0637a08 --- /dev/null +++ b/slack-archive/data/C04QSV0GG23.json @@ -0,0 +1 @@ +[] \ No newline at end of file diff --git a/slack-archive/data/C04THH1V90X.json b/slack-archive/data/C04THH1V90X.json new file mode 100644 index 0000000..0637a08 --- /dev/null +++ b/slack-archive/data/C04THH1V90X.json @@ -0,0 +1 @@ +[] \ No newline at end of file diff --git a/slack-archive/data/C051C93UZK9.json b/slack-archive/data/C051C93UZK9.json new file mode 100644 index 0000000..0637a08 --- /dev/null +++ b/slack-archive/data/C051C93UZK9.json @@ -0,0 +1 @@ +[] \ No newline at end of file diff --git a/slack-archive/data/C055GGUFMHQ.json b/slack-archive/data/C055GGUFMHQ.json new file mode 100644 index 0000000..0637a08 --- /dev/null +++ b/slack-archive/data/C055GGUFMHQ.json @@ -0,0 +1 @@ +[] \ No newline at end of file diff --git a/slack-archive/data/C056YHEU680.json b/slack-archive/data/C056YHEU680.json new file mode 100644 index 0000000..fb1e1a9 --- /dev/null +++ b/slack-archive/data/C056YHEU680.json @@ -0,0 +1,199 @@ +[ + { + "type": "message", + "subtype": "channel_join", + "ts": "1695502057.851259", + "user": "U05TQPZ4R4L", + "text": "<@U05TQPZ4R4L> has joined the channel" + }, + { + "type": "message", + "text": "Some pictures from last night", + "files": [ + { + "id": "F05QC8C72TY", + "created": 1693538277, + "timestamp": 1693538277, + "name": "IMG_2417.jpg", + "title": "IMG_2417", + "mimetype": "image/jpeg", + "filetype": "jpg", + "pretty_type": "JPEG", + "user": "U01DCLP0GU9", + "user_team": "T01CWUYP5AR", + "editable": false, + "size": 1421083, + "mode": "hosted", + "is_external": false, + "external_type": "", + "is_public": true, + "public_url_shared": false, + "display_as_bot": false, + "username": "", + "url_private": "https://files.slack.com/files-pri/T01CWUYP5AR-F05QC8C72TY/img_2417.jpg", + "url_private_download": "https://files.slack.com/files-pri/T01CWUYP5AR-F05QC8C72TY/download/img_2417.jpg", + "media_display_type": "unknown", + "thumb_64": "https://files.slack.com/files-tmb/T01CWUYP5AR-F05QC8C72TY-cee9310a09/img_2417_64.jpg", + "thumb_80": "https://files.slack.com/files-tmb/T01CWUYP5AR-F05QC8C72TY-cee9310a09/img_2417_80.jpg", + "thumb_360": "https://files.slack.com/files-tmb/T01CWUYP5AR-F05QC8C72TY-cee9310a09/img_2417_360.jpg", + "thumb_360_w": 360, + "thumb_360_h": 270, + "thumb_480": "https://files.slack.com/files-tmb/T01CWUYP5AR-F05QC8C72TY-cee9310a09/img_2417_480.jpg", + "thumb_480_w": 480, + "thumb_480_h": 360, + "thumb_160": "https://files.slack.com/files-tmb/T01CWUYP5AR-F05QC8C72TY-cee9310a09/img_2417_160.jpg", + "thumb_720": "https://files.slack.com/files-tmb/T01CWUYP5AR-F05QC8C72TY-cee9310a09/img_2417_720.jpg", + "thumb_720_w": 720, + "thumb_720_h": 540, + "thumb_800": "https://files.slack.com/files-tmb/T01CWUYP5AR-F05QC8C72TY-cee9310a09/img_2417_800.jpg", + "thumb_800_w": 800, + "thumb_800_h": 600, + "thumb_960": "https://files.slack.com/files-tmb/T01CWUYP5AR-F05QC8C72TY-cee9310a09/img_2417_960.jpg", + "thumb_960_w": 960, + "thumb_960_h": 720, + "thumb_1024": "https://files.slack.com/files-tmb/T01CWUYP5AR-F05QC8C72TY-cee9310a09/img_2417_1024.jpg", + "thumb_1024_w": 1024, + "thumb_1024_h": 768, + "original_w": 4032, + "original_h": 3024, + "thumb_tiny": "AwAkADCBGljGMZX06ipllU9QV+vT86ftG8/WpdopMERO4C8jI7GmGNZYyR1AyKc0QJ+XcoPbHBpqRbUdlbt60rDuJLgdMVEpAbJqdLbMYJxmj7MoP3qLBcqiaV2xvY59BUkbskuHJOe5NQKSrArwfWrMcJf5pabBE2w7juckEdM1DIyKxUHB25yakkPlgFcYHrTdiSMGZQcDHNIBkcn7pmV+g5GKcjNFy5PPA3UiR4yEG0HihrdniCu2SvQ0ALCi9cc1N1qOL7tSDrQBXveVQeppydFHam3nSP8A3qcvb60ATDpThTR0pRSGf//Z", + "permalink": "https://openlineage.slack.com/files/U01DCLP0GU9/F05QC8C72TY/img_2417.jpg", + "permalink_public": "https://slack-files.com/T01CWUYP5AR-F05QC8C72TY-6f4277379d", + "is_starred": false, + "has_rich_preview": false, + "file_access": "visible" + }, + { + "id": "F05QJQD2SE7", + "created": 1693538277, + "timestamp": 1693538277, + "name": "IMG_2416.jpg", + "title": "IMG_2416", + "mimetype": "image/jpeg", + "filetype": "jpg", + "pretty_type": "JPEG", + "user": "U01DCLP0GU9", + "user_team": "T01CWUYP5AR", + "editable": false, + "size": 1345466, + "mode": "hosted", + "is_external": false, + "external_type": "", + "is_public": true, + "public_url_shared": false, + "display_as_bot": false, + "username": "", + "url_private": "https://files.slack.com/files-pri/T01CWUYP5AR-F05QJQD2SE7/img_2416.jpg", + "url_private_download": "https://files.slack.com/files-pri/T01CWUYP5AR-F05QJQD2SE7/download/img_2416.jpg", + "media_display_type": "unknown", + "thumb_64": "https://files.slack.com/files-tmb/T01CWUYP5AR-F05QJQD2SE7-e67c68ce7a/img_2416_64.jpg", + "thumb_80": "https://files.slack.com/files-tmb/T01CWUYP5AR-F05QJQD2SE7-e67c68ce7a/img_2416_80.jpg", + "thumb_360": "https://files.slack.com/files-tmb/T01CWUYP5AR-F05QJQD2SE7-e67c68ce7a/img_2416_360.jpg", + "thumb_360_w": 360, + "thumb_360_h": 270, + "thumb_480": "https://files.slack.com/files-tmb/T01CWUYP5AR-F05QJQD2SE7-e67c68ce7a/img_2416_480.jpg", + "thumb_480_w": 480, + "thumb_480_h": 360, + "thumb_160": "https://files.slack.com/files-tmb/T01CWUYP5AR-F05QJQD2SE7-e67c68ce7a/img_2416_160.jpg", + "thumb_720": "https://files.slack.com/files-tmb/T01CWUYP5AR-F05QJQD2SE7-e67c68ce7a/img_2416_720.jpg", + "thumb_720_w": 720, + "thumb_720_h": 540, + "thumb_800": "https://files.slack.com/files-tmb/T01CWUYP5AR-F05QJQD2SE7-e67c68ce7a/img_2416_800.jpg", + "thumb_800_w": 800, + "thumb_800_h": 600, + "thumb_960": "https://files.slack.com/files-tmb/T01CWUYP5AR-F05QJQD2SE7-e67c68ce7a/img_2416_960.jpg", + "thumb_960_w": 960, + "thumb_960_h": 720, + "thumb_1024": "https://files.slack.com/files-tmb/T01CWUYP5AR-F05QJQD2SE7-e67c68ce7a/img_2416_1024.jpg", + "thumb_1024_w": 1024, + "thumb_1024_h": 768, + "original_w": 4032, + "original_h": 3024, + "thumb_tiny": "AwAkADBqysBiRd49R1qQFZExncvt1WmeTtkAL4z0HrTvK3NkEg/r+dKwxSdwCsRn1HQ0ySHbGXzSSxurZLe3TrTnV2iI+chevTmlYLlcYQc/eNIG3nBy3oq/1pwiVnA+bn1NWY4grALxx2p2C5FJNu8s917j6U1EuGGQXwecg1NJArheOn4UmxlQKGIA9KLhYQZkKs7jb9eQR60ku8E7XG0++OKljjAO7AyakZQeoBpXHYz0Z0YktyOnGamhmZpgZABgdqkkiUjgYx6cVAN2/nJ9M+lO4rIuilIpBSmpGJQaKQ0AMccikwNp9qc/akH3G+lMD//Z", + "permalink": "https://openlineage.slack.com/files/U01DCLP0GU9/F05QJQD2SE7/img_2416.jpg", + "permalink_public": "https://slack-files.com/T01CWUYP5AR-F05QJQD2SE7-1062701b11", + "is_starred": false, + "has_rich_preview": false, + "file_access": "visible" + } + ], + "upload": false, + "user": "U01DCLP0GU9", + "ts": "1693538290.446329", + "blocks": [ + { + "type": "rich_text", + "block_id": "SoXJ", + "elements": [ + { + "type": "rich_text_section", + "elements": [ + { + "type": "text", + "text": "Some pictures from last night" + } + ] + } + ] + } + ], + "client_msg_id": "F51EEE94-187B-4A76-9CD0-632E2BE58C1E" + }, + { + "type": "message", + "subtype": "channel_join", + "ts": "1693520941.967439", + "user": "U05Q3HT6PBR", + "text": "<@U05Q3HT6PBR> has joined the channel" + }, + { + "client_msg_id": "1a2569cb-8ea1-45b6-b8a2-aea02fe25d1f", + "type": "message", + "text": "Time: 5:30-8:30 pm\nAddress: 8 California St., San Francisco, CA, seventh floor\nGetting in: someone from Astronomer will be in the lobby to direct you", + "user": "U02LXF3HUN7", + "ts": "1693422775.587509", + "blocks": [ + { + "type": "rich_text", + "block_id": "F3M", + "elements": [ + { + "type": "rich_text_section", + "elements": [ + { + "type": "text", + "text": "Time: 5:30-8:30 pm\nAddress: 8 California St., San Francisco, CA, seventh floor\nGetting in: someone from Astronomer will be in the lobby to direct you" + } + ] + } + ] + } + ], + "team": "T01CWUYP5AR" + }, + { + "client_msg_id": "82e1578c-2f96-4144-a3eb-aa236b28dce2", + "type": "message", + "text": "Adding the venue info in case it’s more convenient than the meetup page:", + "user": "U02LXF3HUN7", + "ts": "1693422678.406409", + "blocks": [ + { + "type": "rich_text", + "block_id": "gcfz", + "elements": [ + { + "type": "rich_text_section", + "elements": [ + { + "type": "text", + "text": "Adding the venue info in case it’s more convenient than the meetup page:" + } + ] + } + ] + } + ], + "team": "T01CWUYP5AR" + } +] \ No newline at end of file diff --git a/slack-archive/data/C05N442RQUA.json b/slack-archive/data/C05N442RQUA.json new file mode 100644 index 0000000..e15d8a4 --- /dev/null +++ b/slack-archive/data/C05N442RQUA.json @@ -0,0 +1,586 @@ +[ + { + "client_msg_id": "6d1c0549-f20e-4c41-9c13-9023d966d51c", + "type": "message", + "text": "Hi, if you’re wondering if you’re in the right place: look for Uncle Tetsu’s Cheesecake nextdoor and for the address (600 Bay St) above the door. The building is an older one (unlike the meeting space itself, which is modern and well-appointed)", + "user": "U02LXF3HUN7", + "ts": "1695068433.208409", + "blocks": [ + { + "type": "rich_text", + "block_id": "1YbUw", + "elements": [ + { + "type": "rich_text_section", + "elements": [ + { + "type": "text", + "text": "Hi, if you’re wondering if you’re in the right place: look for Uncle Tetsu’s Cheesecake nextdoor and for the address (600 Bay St) above the door. The building is an older one (unlike the meeting space itself, which is modern and well-appointed)" + } + ] + } + ] + } + ], + "team": "T01CWUYP5AR" + }, + { + "client_msg_id": "99a7da7d-9233-482c-88c5-ba9bf6ea989f", + "type": "message", + "text": "Looking forward to seeing you on Monday! Here’s the time/place info again for your convenience:\n• Date: 9/18\n• Time: 5-8:00 PM ET\n• Place: Canarts, 600 Bay St., #410 (around the corner from the Airflow Summit venue)\n• Venue phone: \n• Meetup page with more info and signup: \nPlease send a message if you find yourself stuck in the lobby, etc.", + "user": "U02LXF3HUN7", + "ts": "1694794649.751439", + "blocks": [ + { + "type": "rich_text", + "block_id": "l3F59", + "elements": [ + { + "type": "rich_text_section", + "elements": [ + { + "type": "text", + "text": "Looking forward to seeing you on Monday! Here’s the time/place info again for your convenience:\n" + } + ] + }, + { + "type": "rich_text_list", + "elements": [ + { + "type": "rich_text_section", + "elements": [ + { + "type": "text", + "text": "Date: 9/18" + } + ] + }, + { + "type": "rich_text_section", + "elements": [ + { + "type": "text", + "text": "Time: 5-8:00 PM ET" + } + ] + }, + { + "type": "rich_text_section", + "elements": [ + { + "type": "text", + "text": "Place: Canarts, 600 Bay St., #410 (around the corner from the Airflow Summit venue)" + } + ] + }, + { + "type": "rich_text_section", + "elements": [ + { + "type": "text", + "text": "Venue phone: " + }, + { + "type": "link", + "url": "tel:4168052286", + "text": "416-805-2286" + } + ] + }, + { + "type": "rich_text_section", + "elements": [ + { + "type": "text", + "text": "Meetup page with more info and signup: " + }, + { + "type": "link", + "url": "https://www.meetup.com/openlineage/events/295488014/?utm_medium=referral&utm_campaign=share-btn_savedevents_share_modal&utm_source=link" + } + ] + } + ], + "style": "bullet", + "indent": 0, + "border": 0 + }, + { + "type": "rich_text_section", + "elements": [ + { + "type": "text", + "text": "Please send a message if you find yourself stuck in the lobby, etc." + } + ] + } + ] + } + ], + "team": "T01CWUYP5AR", + "attachments": [ + { + "from_url": "https://www.meetup.com/openlineage/events/295488014/?utm_medium=referral&utm_campaign=share-btn_savedevents_share_modal&utm_source=link", + "image_url": "https://secure.meetupstatic.com/photos/event/5/4/2/d/600_515181549.jpeg", + "image_width": 600, + "image_height": 338, + "image_bytes": 16248, + "service_icon": "https://secure.meetupstatic.com/next/images/general/m_swarm_120x120.png", + "id": 1, + "original_url": "https://www.meetup.com/openlineage/events/295488014/?utm_medium=referral&utm_campaign=share-btn_savedevents_share_modal&utm_source=link", + "fallback": "Meetup: Toronto OpenLineage Meetup at Airflow Summit, Mon, Sep 18, 2023, 5:00 PM | Meetup", + "text": "Data engineers and pipeline managers know that producing data lineage – end-to-end pipeline metadata instrumented at runtime or parsed at design time – is a heavy lift with", + "title": "Toronto OpenLineage Meetup at Airflow Summit, Mon, Sep 18, 2023, 5:00 PM | Meetup", + "title_link": "https://www.meetup.com/openlineage/events/295488014/?utm_medium=referral&utm_campaign=share-btn_savedevents_share_modal&utm_source=link", + "service_name": "Meetup" + } + ], + "reactions": [ + { + "name": "raised_hands", + "users": [ + "U01RA9B5GG2" + ], + "count": 1 + } + ] + }, + { + "type": "message", + "subtype": "channel_join", + "ts": "1694787071.646859", + "user": "U05SXDWVA7K", + "text": "<@U05SXDWVA7K> has joined the channel" + }, + { + "client_msg_id": "6cfdc82a-896e-4963-9187-717456008417", + "type": "message", + "text": "\nIt’s hard to believe this is happening in just one week! Here’s the updated agenda:\n1. *Intros*\n2. *Evolution of spec presentation/discussion (project background/history)*\n3. *State of the community*\n4. *Integrating OpenLineage with (by special guests & )*\n5. *Spark/Column lineage update*\n6. *Airflow Provider update*\n7. *Roadmap Discussion*\n8. *Action items review/next steps*\nFind the details and RSVP .", + "user": "U02LXF3HUN7", + "ts": "1694441637.116609", + "blocks": [ + { + "type": "rich_text", + "block_id": "Go5ub", + "elements": [ + { + "type": "rich_text_section", + "elements": [ + { + "type": "broadcast", + "range": "channel" + }, + { + "type": "text", + "text": "\nIt’s hard to believe this is happening in just one week! Here’s the updated agenda:\n" + } + ] + }, + { + "type": "rich_text_list", + "elements": [ + { + "type": "rich_text_section", + "elements": [ + { + "type": "text", + "text": "Intros", + "style": { + "bold": true + } + } + ] + }, + { + "type": "rich_text_section", + "elements": [ + { + "type": "text", + "text": "Evolution of spec presentation/discussion (project background/history)", + "style": { + "bold": true + } + } + ] + }, + { + "type": "rich_text_section", + "elements": [ + { + "type": "text", + "text": "State of the community", + "style": { + "bold": true + } + } + ] + }, + { + "type": "rich_text_section", + "elements": [ + { + "type": "text", + "text": "Integrating OpenLineage with ", + "style": { + "bold": true + } + }, + { + "type": "link", + "url": "https://metaphor.io/", + "text": "Metaphor", + "style": { + "bold": true + } + }, + { + "type": "text", + "text": " (by special guests ", + "style": { + "bold": true + } + }, + { + "type": "link", + "url": "https://www.linkedin.com/in/yeliu84/", + "text": "Ye", + "style": { + "bold": true + } + }, + { + "type": "text", + "text": " & ", + "style": { + "bold": true + } + }, + { + "type": "link", + "url": "https://www.linkedin.com/in/ivanperepelitca/", + "text": "Ivan", + "style": { + "bold": true + } + }, + { + "type": "text", + "text": ")", + "style": { + "bold": true + } + } + ] + }, + { + "type": "rich_text_section", + "elements": [ + { + "type": "text", + "text": "Spark/Column lineage update", + "style": { + "bold": true + } + } + ] + }, + { + "type": "rich_text_section", + "elements": [ + { + "type": "text", + "text": "Airflow Provider update", + "style": { + "bold": true + } + } + ] + }, + { + "type": "rich_text_section", + "elements": [ + { + "type": "text", + "text": "Roadmap Discussion", + "style": { + "bold": true + } + } + ] + }, + { + "type": "rich_text_section", + "elements": [ + { + "type": "text", + "text": "Action items review/next steps", + "style": { + "bold": true + } + } + ] + } + ], + "style": "ordered", + "indent": 0, + "border": 0 + }, + { + "type": "rich_text_section", + "elements": [ + { + "type": "text", + "text": "Find the details and RSVP " + }, + { + "type": "link", + "url": "https://www.meetup.com/openlineage/events/295488014/?utm_medium=referral&utm_campaign=share-btn_savedevents_share_modal&utm_source=link", + "text": "here" + }, + { + "type": "text", + "text": "." + } + ] + } + ] + } + ], + "team": "T01CWUYP5AR", + "attachments": [ + { + "image_url": "https://static.metaphor.io/preview.jpg", + "image_width": 719, + "image_height": 378, + "image_bytes": 122301, + "from_url": "https://metaphor.io/", + "id": 1, + "original_url": "https://metaphor.io/", + "fallback": "Metaphor - The Social Platform for Data", + "text": "Making Data Actionable, At Scale - Designed for data teams building cloud-native, self-service data platforms for their business users. Explore our Data Governance, Data Lineage, Data Discovery, and Data Trust capabilities today.", + "title": "Metaphor - The Social Platform for Data", + "title_link": "https://metaphor.io/", + "service_name": "metaphor.io" + }, + { + "image_url": "https://secure.meetupstatic.com/photos/event/5/4/2/d/600_515181549.jpeg", + "image_width": 600, + "image_height": 338, + "image_bytes": 16248, + "from_url": "https://www.meetup.com/openlineage/events/295488014/?utm_medium=referral&utm_campaign=share-btn_savedevents_share_modal&utm_source=link", + "service_icon": "https://secure.meetupstatic.com/next/images/general/m_swarm_120x120.png", + "id": 2, + "original_url": "https://www.meetup.com/openlineage/events/295488014/?utm_medium=referral&utm_campaign=share-btn_savedevents_share_modal&utm_source=link", + "fallback": "Meetup: Toronto OpenLineage Meetup at Airflow Summit, Mon, Sep 18, 2023, 5:00 PM | Meetup", + "text": "Data engineers and pipeline managers know that producing data lineage – end-to-end pipeline metadata instrumented at runtime or parsed at design time – is a heavy lift with", + "title": "Toronto OpenLineage Meetup at Airflow Summit, Mon, Sep 18, 2023, 5:00 PM | Meetup", + "title_link": "https://www.meetup.com/openlineage/events/295488014/?utm_medium=referral&utm_campaign=share-btn_savedevents_share_modal&utm_source=link", + "service_name": "Meetup" + } + ], + "reactions": [ + { + "name": "raised_hands", + "users": [ + "U01DCMDFHBK" + ], + "count": 1 + } + ] + }, + { + "client_msg_id": "bdb67b53-48b3-4864-93e4-b650780c786d", + "type": "message", + "text": "Most OpenLineage regular contributors will be there. It will be fun to be all in person. Everyone is encouraged to join", + "user": "U01DCLP0GU9", + "ts": "1693624251.155569", + "blocks": [ + { + "type": "rich_text", + "block_id": "=fiF4", + "elements": [ + { + "type": "rich_text_section", + "elements": [ + { + "type": "text", + "text": "Most OpenLineage regular contributors will be there. It will be fun to be all in person. Everyone is encouraged to join" + } + ] + } + ] + } + ], + "team": "T01CWUYP5AR", + "reactions": [ + { + "name": "raised_hands", + "users": [ + "U01HNKK4XAM", + "U01RA9B5GG2", + "U02MK6YNAQ5", + "U02LXF3HUN7", + "U01DCMDFHBK", + "U05KKM07PJP" + ], + "count": 6 + } + ] + }, + { + "client_msg_id": "30dcc90d-e303-46ad-821f-b3ae4b823497", + "type": "message", + "text": "really looking forward to meeting all of you in Toronto!!", + "user": "U01HNKK4XAM", + "ts": "1692984822.264569", + "blocks": [ + { + "type": "rich_text", + "block_id": "e1s", + "elements": [ + { + "type": "rich_text_section", + "elements": [ + { + "type": "text", + "text": "really looking forward to meeting all of you in Toronto!!" + } + ] + } + ] + } + ], + "team": "T01CWUYP5AR" + }, + { + "client_msg_id": "c4139590-6c9a-4a20-99f3-041ab782263e", + "type": "message", + "text": "Some belated updates on this in case you’re not aware:\n• Date: 9/18\n• Time: 5-8:00 PM ET\n• Place: Canarts, 600 Bay St., #410 (around the corner from the Airflow Summit venue)\n• Venue phone: \n• Meetup for more info and to sign up: ", + "user": "U02LXF3HUN7", + "ts": "1692984607.290939", + "blocks": [ + { + "type": "rich_text", + "block_id": "yvbeN", + "elements": [ + { + "type": "rich_text_section", + "elements": [ + { + "type": "text", + "text": "Some belated updates on this in case you’re not aware:\n" + } + ] + }, + { + "type": "rich_text_list", + "elements": [ + { + "type": "rich_text_section", + "elements": [ + { + "type": "text", + "text": "Date: 9/18" + } + ] + }, + { + "type": "rich_text_section", + "elements": [ + { + "type": "text", + "text": "Time: 5-8:00 PM ET" + } + ] + }, + { + "type": "rich_text_section", + "elements": [ + { + "type": "text", + "text": "Place: Canarts, 600 Bay St., #410 (around the corner from the Airflow Summit venue)" + } + ] + }, + { + "type": "rich_text_section", + "elements": [ + { + "type": "text", + "text": "Venue phone: " + }, + { + "type": "link", + "url": "tel:4168052286", + "text": "416-805-2286" + } + ] + }, + { + "type": "rich_text_section", + "elements": [ + { + "type": "text", + "text": "Meetup for more info and to sign up: " + }, + { + "type": "link", + "url": "https://www.meetup.com/openlineage/events/295488014/?utm_medium=referral&utm_campaign=share-btn_savedevents_share_modal&utm_source=link" + } + ] + } + ], + "style": "bullet", + "indent": 0, + "border": 0 + } + ] + } + ], + "team": "T01CWUYP5AR", + "edited": { + "user": "U02LXF3HUN7", + "ts": "1694786010.000000" + }, + "attachments": [ + { + "from_url": "https://www.meetup.com/openlineage/events/295488014/?utm_medium=referral&utm_campaign=share-btn_savedevents_share_modal&utm_source=link", + "image_url": "https://secure.meetupstatic.com/photos/event/5/4/2/d/600_515181549.jpeg", + "image_width": 600, + "image_height": 338, + "image_bytes": 16248, + "service_icon": "https://secure.meetupstatic.com/next/images/general/m_swarm_120x120.png", + "id": 1, + "original_url": "https://www.meetup.com/openlineage/events/295488014/?utm_medium=referral&utm_campaign=share-btn_savedevents_share_modal&utm_source=link", + "fallback": "Meetup: Toronto OpenLineage Meetup at Airflow Summit, Mon, Sep 18, 2023, 2:00 PM | Meetup", + "text": "Data engineers and pipeline managers know that producing data lineage – end-to-end pipeline metadata instrumented at runtime or parsed at design time – is a heavy lift with", + "title": "Toronto OpenLineage Meetup at Airflow Summit, Mon, Sep 18, 2023, 2:00 PM | Meetup", + "title_link": "https://www.meetup.com/openlineage/events/295488014/?utm_medium=referral&utm_campaign=share-btn_savedevents_share_modal&utm_source=link", + "service_name": "Meetup" + } + ], + "reactions": [ + { + "name": "tada", + "users": [ + "U01HNKK4XAM", + "U01RA9B5GG2", + "U02MK6YNAQ5", + "U01DCMDFHBK" + ], + "count": 4 + }, + { + "name": "raised_hands", + "users": [ + "U01HNKK4XAM", + "U01RA9B5GG2", + "U02MK6YNAQ5", + "U01DCMDFHBK" + ], + "count": 4 + } + ] + } +] \ No newline at end of file diff --git a/slack-archive/data/C05PD7VJ52S.json b/slack-archive/data/C05PD7VJ52S.json new file mode 100644 index 0000000..0968ddc --- /dev/null +++ b/slack-archive/data/C05PD7VJ52S.json @@ -0,0 +1,382 @@ +[ + { + "type": "message", + "subtype": "channel_join", + "ts": "1693512257.995209", + "user": "U05QHG1NJ8J", + "text": "<@U05QHG1NJ8J> has joined the channel" + }, + { + "type": "message", + "subtype": "channel_join", + "ts": "1692984861.392349", + "user": "U01RA9B5GG2", + "text": "<@U01RA9B5GG2> has joined the channel", + "inviter": "U01HNKK4XAM" + }, + { + "client_msg_id": "c1976b81-2c3c-47ac-a0a9-1563847ea465", + "type": "message", + "text": "Yes, hope so! Thank you for your interest in joining a meetup!", + "user": "U02LXF3HUN7", + "ts": "1692983998.092189", + "blocks": [ + { + "type": "rich_text", + "block_id": "7/R", + "elements": [ + { + "type": "rich_text_section", + "elements": [ + { + "type": "text", + "text": "Yes, hope so! Thank you for your interest in joining a meetup!" + } + ] + } + ] + } + ], + "team": "T01CWUYP5AR" + }, + { + "client_msg_id": "90b6a8f3-45be-4769-9bb3-8099697713b9", + "type": "message", + "text": "hopefully meet you soon in London", + "user": "U05HK41VCH1", + "ts": "1692982281.255129", + "blocks": [ + { + "type": "rich_text", + "block_id": "9eK", + "elements": [ + { + "type": "rich_text_section", + "elements": [ + { + "type": "text", + "text": "hopefully meet you soon in London" + } + ] + } + ] + } + ], + "team": "T01CWUYP5AR" + }, + { + "client_msg_id": "d981a3fe-45e4-4047-bc62-0fb1ac36d8c6", + "type": "message", + "text": "Thanks Michael for starting this channel", + "user": "U05HK41VCH1", + "ts": "1692982268.668049", + "blocks": [ + { + "type": "rich_text", + "block_id": "t6DYs", + "elements": [ + { + "type": "rich_text_section", + "elements": [ + { + "type": "text", + "text": "Thanks Michael for starting this channel" + } + ] + } + ] + } + ], + "team": "T01CWUYP5AR" + }, + { + "client_msg_id": "2c707994-db70-4e93-8f18-b76f192b1b66", + "type": "message", + "text": "yes absolutely will give you an answer by Monday", + "user": "U05J5GRKY10", + "ts": "1692979169.558489", + "blocks": [ + { + "type": "rich_text", + "block_id": "U97Fk", + "elements": [ + { + "type": "rich_text_section", + "elements": [ + { + "type": "text", + "text": "yes absolutely will give you an answer by Monday" + } + ] + } + ] + } + ], + "team": "T01CWUYP5AR", + "reactions": [ + { + "name": "+1", + "users": [ + "U02LXF3HUN7" + ], + "count": 1 + } + ] + }, + { + "client_msg_id": "a309f1f6-375a-4b9d-b024-a18d82bf7cb4", + "type": "message", + "text": "OK! Would you please let me know when you know, and we’ll go from there?", + "user": "U02LXF3HUN7", + "ts": "1692979126.425069", + "blocks": [ + { + "type": "rich_text", + "block_id": "yKFbq", + "elements": [ + { + "type": "rich_text_section", + "elements": [ + { + "type": "text", + "text": "OK! Would you please let me know when you know, and we’ll go from there?" + } + ] + } + ] + } + ], + "team": "T01CWUYP5AR" + }, + { + "client_msg_id": "b23a7e78-ee51-4fc0-af50-a95e8352a4d8", + "type": "message", + "text": "and if that not the case i can provide personal space", + "user": "U05J5GRKY10", + "ts": "1692979080.273399", + "blocks": [ + { + "type": "rich_text", + "block_id": "/9JOs", + "elements": [ + { + "type": "rich_text_section", + "elements": [ + { + "type": "text", + "text": "and if that not the case i can provide personal space" + } + ] + } + ] + } + ], + "team": "T01CWUYP5AR" + }, + { + "client_msg_id": "489d9333-80ab-4848-9d60-f0ceec0c0214", + "type": "message", + "text": "I am pretty sure you can use our 6point6 offices or at least part of it", + "user": "U05J5GRKY10", + "ts": "1692979067.390839", + "blocks": [ + { + "type": "rich_text", + "block_id": "RoGJB", + "elements": [ + { + "type": "rich_text_section", + "elements": [ + { + "type": "text", + "text": "I am pretty sure you can use our 6point6 offices or at least part of it" + } + ] + } + ] + } + ], + "team": "T01CWUYP5AR" + }, + { + "client_msg_id": "b6a2f006-3fe7-4990-a727-671422b36d3d", + "type": "message", + "text": "I will have to confirm but 99% yes", + "user": "U05J5GRKY10", + "ts": "1692978958.692039", + "blocks": [ + { + "type": "rich_text", + "block_id": "wlacd", + "elements": [ + { + "type": "rich_text_section", + "elements": [ + { + "type": "text", + "text": "I will have to confirm but 99% yes" + } + ] + } + ] + } + ], + "team": "T01CWUYP5AR" + }, + { + "client_msg_id": "c1f7cbd7-f90e-4afc-a7d8-2f0b00323b55", + "type": "message", + "text": "Great! Do you happen to have space we could use?", + "user": "U02LXF3HUN7", + "ts": "1692978871.220909", + "blocks": [ + { + "type": "rich_text", + "block_id": "kTLh", + "elements": [ + { + "type": "rich_text_section", + "elements": [ + { + "type": "text", + "text": "Great! Do you happen to have space we could use?" + } + ] + } + ] + } + ], + "team": "T01CWUYP5AR" + }, + { + "client_msg_id": "e2a13328-0735-43f7-a2e8-22a9ed79787f", + "type": "message", + "text": "thats perfect !", + "user": "U05J5GRKY10", + "ts": "1692978852.583609", + "blocks": [ + { + "type": "rich_text", + "block_id": "Cm8bi", + "elements": [ + { + "type": "rich_text_section", + "elements": [ + { + "type": "text", + "text": "thats perfect !" + } + ] + } + ] + } + ], + "team": "T01CWUYP5AR" + }, + { + "client_msg_id": "2bf7b6c8-44fc-4ed4-b19f-f115bcbed49c", + "type": "message", + "text": "Hi George, nice to meet you. Thanks for asking about future meetups. Would November be too soon, or what’s a good timeframe for you all?", + "user": "U02LXF3HUN7", + "ts": "1692978834.508549", + "blocks": [ + { + "type": "rich_text", + "block_id": "hQj1", + "elements": [ + { + "type": "rich_text_section", + "elements": [ + { + "type": "text", + "text": "Hi George, nice to meet you. Thanks for asking about future meetups. Would November be too soon, or what’s a good timeframe for you all?" + } + ] + } + ] + } + ], + "team": "T01CWUYP5AR" + }, + { + "client_msg_id": "f3a13e65-b8e8-4d7d-acaa-b0b5d466029a", + "type": "message", + "text": "thanks so much !", + "user": "U05J5GRKY10", + "ts": "1692978816.416879", + "blocks": [ + { + "type": "rich_text", + "block_id": "q=t=", + "elements": [ + { + "type": "rich_text_section", + "elements": [ + { + "type": "text", + "text": "thanks so much !" + } + ] + } + ] + } + ], + "team": "T01CWUYP5AR" + }, + { + "client_msg_id": "9e029cb5-0c45-452f-af4d-03958ed1b279", + "type": "message", + "text": "Hi Michael", + "user": "U05J5GRKY10", + "ts": "1692978769.711289", + "blocks": [ + { + "type": "rich_text", + "block_id": "yR4", + "elements": [ + { + "type": "rich_text_section", + "elements": [ + { + "type": "text", + "text": "Hi Michael" + } + ] + } + ] + } + ], + "team": "T01CWUYP5AR" + }, + { + "type": "message", + "subtype": "channel_join", + "ts": "1692978760.556819", + "user": "U05HK41VCH1", + "text": "<@U05HK41VCH1> has joined the channel", + "inviter": "U02LXF3HUN7" + }, + { + "type": "message", + "subtype": "channel_join", + "ts": "1692978735.461499", + "user": "U01HNKK4XAM", + "text": "<@U01HNKK4XAM> has joined the channel", + "inviter": "U02LXF3HUN7" + }, + { + "type": "message", + "subtype": "channel_join", + "ts": "1692978735.367849", + "user": "U05J5GRKY10", + "text": "<@U05J5GRKY10> has joined the channel", + "inviter": "U02LXF3HUN7" + }, + { + "type": "message", + "subtype": "channel_join", + "ts": "1692978725.076179", + "user": "U02LXF3HUN7", + "text": "<@U02LXF3HUN7> has joined the channel" + } +] \ No newline at end of file diff --git a/slack-archive/data/C05U3UC85LM.json b/slack-archive/data/C05U3UC85LM.json new file mode 100644 index 0000000..f8d81b0 --- /dev/null +++ b/slack-archive/data/C05U3UC85LM.json @@ -0,0 +1,336 @@ +[ + { + "type": "message", + "subtype": "channel_join", + "ts": "1697734944.171989", + "user": "U0620HU51HA", + "text": "<@U0620HU51HA> has joined the channel" + }, + { + "client_msg_id": "fe3c2887-f751-445f-a711-034c74a1a00f", + "type": "message", + "text": "<@U05U9K21LSG> would be great if we could get your eyes on this PR: ", + "user": "U01HNKK4XAM", + "ts": "1697201275.793679", + "blocks": [ + { + "type": "rich_text", + "block_id": "+lkFZ", + "elements": [ + { + "type": "rich_text_section", + "elements": [ + { + "type": "user", + "user_id": "U05U9K21LSG" + }, + { + "type": "text", + "text": " would be great if we could get your eyes on this PR: " + }, + { + "type": "link", + "url": "https://github.com/OpenLineage/OpenLineage/pull/2134" + } + ] + } + ] + } + ], + "team": "T01CWUYP5AR", + "attachments": [ + { + "id": 1, + "footer_icon": "https://slack.github.com/static/img/favicon-neutral.png", + "ts": 1695798428, + "color": "36a64f", + "bot_id": "B01VA0FB340", + "app_unfurl_url": "https://github.com/OpenLineage/OpenLineage/pull/2134", + "is_app_unfurl": true, + "app_id": "A01BP7R4KNY", + "fallback": "#2134 Support more recent versions of GX", + "text": "Limit upper GX version to <0.16.0.\n\nReopening accidentally closed by me.\n\nKudos to .\n\n*Checklist*\n\n☑︎ You've your work\n☑︎ Your pull request title follows our \n☑︎ Your changes are accompanied by tests (_if relevant_)\n☑︎ Your change contains a and is self-contained\n☐ You've updated any relevant documentation (_if relevant_)\n☐ Your comment includes a one-liner for the changelog about the specific purpose of the change (_if necessary_)\n☐ You've versioned the core OpenLineage model or facets according to (_if relevant_)\n☐ You've added a to source files (_if relevant_)\n\n* * *\n\nSPDX-License-Identifier: Apache-2.0 \nCopyright 2018-2023 contributors to the OpenLineage project", + "title": "#2134 Support more recent versions of GX", + "title_link": "https://github.com/OpenLineage/OpenLineage/pull/2134", + "footer": "", + "fields": [ + { + "value": "integration/great-expectations, common", + "title": "Labels", + "short": true + }, + { + "value": "6", + "title": "Comments", + "short": true + } + ], + "mrkdwn_in": [ + "text" + ] + } + ], + "thread_ts": "1697201275.793679", + "reply_count": 1, + "reply_users_count": 1, + "latest_reply": "1697223637.244249", + "reply_users": [ + "U05U9K21LSG" + ], + "is_locked": false, + "subscribed": false, + "replies": [ + { + "client_msg_id": "ce4bbaf7-c460-42c8-8cdc-787411c365cc", + "type": "message", + "text": "I'm a bit slammed today but can look on Tuesday.", + "user": "U05U9K21LSG", + "ts": "1697223637.244249", + "blocks": [ + { + "type": "rich_text", + "block_id": "fW0Bl", + "elements": [ + { + "type": "rich_text_section", + "elements": [ + { + "type": "text", + "text": "I'm a bit slammed today but can look on Tuesday." + } + ] + } + ] + } + ], + "team": "T01CWUYP5AR", + "thread_ts": "1697201275.793679", + "parent_user_id": "U01HNKK4XAM", + "reactions": [ + { + "name": "white_check_mark", + "users": [ + "U01HNKK4XAM" + ], + "count": 1 + } + ] + } + ] + }, + { + "client_msg_id": "676cb6ce-3f9e-405c-b2d7-21d716026b35", + "type": "message", + "text": "Just seeing this, we had a company holiday yesterday. Yes, fluent data sources are our new way of connecting to data and the older \"block-style\" is deprecated and will be removed when we cut 0.18.0. I'm not sure of the timing of that but likely in the next couple months.", + "user": "U05U9K21LSG", + "ts": "1696979522.397239", + "blocks": [ + { + "type": "rich_text", + "block_id": "z1dsG", + "elements": [ + { + "type": "rich_text_section", + "elements": [ + { + "type": "text", + "text": "Just seeing this, we had a company holiday yesterday. Yes, fluent data sources are our new way of connecting to data and the older \"block-style\" is deprecated and will be removed when we cut 0.18.0. I'm not sure of the timing of that but likely in the next couple months." + } + ] + } + ] + } + ], + "team": "T01CWUYP5AR", + "reactions": [ + { + "name": "+1", + "users": [ + "U02S6F54MAB" + ], + "count": 1 + } + ] + }, + { + "client_msg_id": "ce3473ef-55b7-4544-b100-f84b90dedcc0", + "type": "message", + "text": "<@U05U9929K3N> <@U05U9K21LSG> ^^", + "user": "U01HNKK4XAM", + "ts": "1696860879.713919", + "blocks": [ + { + "type": "rich_text", + "block_id": "VrCEQ", + "elements": [ + { + "type": "rich_text_section", + "elements": [ + { + "type": "user", + "user_id": "U05U9929K3N" + }, + { + "type": "text", + "text": " " + }, + { + "type": "user", + "user_id": "U05U9K21LSG" + }, + { + "type": "text", + "text": " ^^" + } + ] + } + ] + } + ], + "team": "T01CWUYP5AR" + }, + { + "client_msg_id": "a458875f-f440-48fa-8583-cdd43dd35eae", + "type": "message", + "text": "Hello guys! I’ve been looking recently into changes in GX.\n\nis this the major change you’d like to introduce in OL<-> GX?", + "user": "U02S6F54MAB", + "ts": "1696852012.119669", + "blocks": [ + { + "type": "rich_text", + "block_id": "77ma2", + "elements": [ + { + "type": "rich_text_section", + "elements": [ + { + "type": "text", + "text": "Hello guys! I’ve been looking recently into changes in GX.\n" + }, + { + "type": "link", + "url": "https://greatexpectations.io/blog/the-fluent-way-to-connect-to-data-sources-in-gx/" + }, + { + "type": "text", + "text": "\nis this the major change you’d like to introduce in OL<-> GX?" + } + ] + } + ] + } + ], + "team": "T01CWUYP5AR", + "attachments": [ + { + "image_url": "https://images.ctfassets.net/ycwst8v1r2x5/5Ec3CoEgrMo6EtNF64XL4Q/9ba1176e49c83e6dfaeafe089c3f326c/fluent_cover_card.png", + "image_width": 1560, + "image_height": 1040, + "image_bytes": 1828778, + "from_url": "https://greatexpectations.io/blog/the-fluent-way-to-connect-to-data-sources-in-gx/", + "service_icon": "https://greatexpectations.io/icons/icon-48x48.png?v=62b42a6c11e70607aa006eb9abc523ed", + "id": 1, + "original_url": "https://greatexpectations.io/blog/the-fluent-way-to-connect-to-data-sources-in-gx/", + "fallback": "The fluent way to connect to data sources in GX", + "text": "Creating a Datasource just got (much) easier", + "title": "The fluent way to connect to data sources in GX", + "title_link": "https://greatexpectations.io/blog/the-fluent-way-to-connect-to-data-sources-in-gx/", + "service_name": "greatexpectations.io" + } + ] + }, + { + "type": "message", + "subtype": "channel_join", + "ts": "1695916771.601439", + "user": "U05U9K21LSG", + "text": "<@U05U9K21LSG> has joined the channel" + }, + { + "client_msg_id": "854a6cc9-f46f-4955-a845-f33242ac4025", + "type": "message", + "text": "<@U05U9929K3N> it was great meeting earlier, looking forward to collaborating on this!", + "user": "U01HNKK4XAM", + "ts": "1695840258.594299", + "blocks": [ + { + "type": "rich_text", + "block_id": "vMZ5h", + "elements": [ + { + "type": "rich_text_section", + "elements": [ + { + "type": "user", + "user_id": "U05U9929K3N" + }, + { + "type": "text", + "text": " it was great meeting earlier, looking forward to collaborating on this!" + } + ] + } + ] + } + ], + "team": "T01CWUYP5AR", + "reactions": [ + { + "name": "heavy_plus_sign", + "users": [ + "U05U9929K3N", + "U02S6F54MAB" + ], + "count": 2 + } + ] + }, + { + "type": "message", + "subtype": "channel_join", + "ts": "1695836477.653589", + "user": "U01RA9B5GG2", + "text": "<@U01RA9B5GG2> has joined the channel", + "inviter": "U02LXF3HUN7" + }, + { + "type": "message", + "subtype": "channel_join", + "ts": "1695836303.953919", + "user": "U02S6F54MAB", + "text": "<@U02S6F54MAB> has joined the channel", + "inviter": "U02LXF3HUN7" + }, + { + "type": "message", + "subtype": "channel_join", + "ts": "1695836303.827639", + "user": "U01HNKK4XAM", + "text": "<@U01HNKK4XAM> has joined the channel", + "inviter": "U02LXF3HUN7" + }, + { + "type": "message", + "subtype": "channel_join", + "ts": "1695836303.727049", + "user": "U05U9929K3N", + "text": "<@U05U9929K3N> has joined the channel", + "inviter": "U02LXF3HUN7" + }, + { + "type": "message", + "subtype": "channel_join", + "ts": "1695836283.617309", + "user": "U02LXF3HUN7", + "text": "<@U02LXF3HUN7> has joined the channel", + "reactions": [ + { + "name": "tada", + "users": [ + "U01HNKK4XAM" + ], + "count": 1 + } + ] + } +] \ No newline at end of file diff --git a/slack-archive/data/C065PQ4TL8K.json b/slack-archive/data/C065PQ4TL8K.json new file mode 100644 index 0000000..9920c44 --- /dev/null +++ b/slack-archive/data/C065PQ4TL8K.json @@ -0,0 +1,2466 @@ +[ + { + "client_msg_id": "094cab1a-de83-45a2-96ff-8efdaf41e09c", + "type": "message", + "text": "Maybe move today's meeting earlier, since no one from west coast is joining? <@U01HNKK4XAM>", + "user": "U01RA9B5GG2", + "ts": "1700562211.366219", + "blocks": [ + { + "type": "rich_text", + "block_id": "4la1W", + "elements": [ + { + "type": "rich_text_section", + "elements": [ + { + "type": "text", + "text": "Maybe move today's meeting earlier, since no one from west coast is joining? " + }, + { + "type": "user", + "user_id": "U01HNKK4XAM" + } + ] + } + ] + } + ], + "team": "T01CWUYP5AR" + }, + { + "client_msg_id": "c67600bf-5ef9-411c-a0d4-bd4dabef0b3c", + "type": "message", + "text": "I’m off on vacation. See you in a week", + "user": "U01DCLP0GU9", + "ts": "1700272614.735719", + "blocks": [ + { + "type": "rich_text", + "block_id": "fzif9", + "elements": [ + { + "type": "rich_text_section", + "elements": [ + { + "type": "text", + "text": "I’m off on vacation. See you in a week" + } + ] + } + ] + } + ], + "team": "T01CWUYP5AR", + "reactions": [ + { + "name": "heart", + "users": [ + "U02S6F54MAB", + "U01RA9B5GG2", + "U02MK6YNAQ5", + "U01HNKK4XAM", + "U053LLVTHRN" + ], + "count": 5 + } + ] + }, + { + "client_msg_id": "090c8b69-40d1-44f5-aaf6-89200841114e", + "type": "message", + "text": "just searching for OpenLineage in the Datahub code base. They have an “interesting” approach? ", + "user": "U01DCLP0GU9", + "ts": "1700246539.228259", + "blocks": [ + { + "type": "rich_text", + "block_id": "c5Ci8", + "elements": [ + { + "type": "rich_text_section", + "elements": [ + { + "type": "text", + "text": "just searching for OpenLineage in the Datahub code base. They have an “interesting” approach? " + }, + { + "type": "link", + "url": "https://github.com/datahub-project/datahub/blob/2b0811b9875d7d7ea11fb01d0157a21fdd67f020/metadata-ingestion-modules/airflow-plugin/src/datahub_airflow_plugin/_extractors.py#L14", + "text": "https://github.com/datahub-project/datahub/blob/2b0811b9875d7d7ea11fb01d0157a21fdd[…]odules/airflow-plugin/src/datahub_airflow_plugin/_extractors.py" + } + ] + } + ] + } + ], + "team": "T01CWUYP5AR", + "attachments": [ + { + "id": 1, + "footer_icon": "https://slack.github.com/static/img/favicon-neutral.png", + "color": "24292f", + "bot_id": "B01VA0FB340", + "app_unfurl_url": "https://github.com/datahub-project/datahub/blob/2b0811b9875d7d7ea11fb01d0157a21fdd67f020/metadata-ingestion-modules/airflow-plugin/src/datahub_airflow_plugin/_extractors.py#L14", + "is_app_unfurl": true, + "app_id": "A01BP7R4KNY", + "fallback": "", + "text": "```\nfrom openlineage.airflow.extractors import BaseExtractor\n```", + "title": "", + "footer": "", + "mrkdwn_in": [ + "text" + ] + } + ], + "thread_ts": "1700246539.228259", + "reply_count": 7, + "reply_users_count": 3, + "latest_reply": "1700467087.549929", + "reply_users": [ + "U01DCLP0GU9", + "U02S6F54MAB", + "U02MK6YNAQ5" + ], + "is_locked": false, + "subscribed": false, + "replies": [ + { + "client_msg_id": "26a6b41b-422e-42ba-9f3e-322422a370a7", + "type": "message", + "text": "It looks like the datahub airflow plugin uses OL. but turns it off\n\n```disable_openlineage_plugin\ttrue\tDisable the OpenLineage plugin to avoid duplicative processing.```\nThey reuse the extractors but then “patch” the behavior.", + "user": "U01DCLP0GU9", + "ts": "1700246841.718379", + "blocks": [ + { + "type": "rich_text", + "block_id": "DREhv", + "elements": [ + { + "type": "rich_text_section", + "elements": [ + { + "type": "text", + "text": "It looks like the datahub airflow plugin uses OL. but turns it off\n" + }, + { + "type": "link", + "url": "https://github.com/datahub-project/datahub/blob/2b0811b9875d7d7ea11fb01d0157a21fdd67f020/docs/lineage/airflow.md" + }, + { + "type": "text", + "text": "\n" + } + ] + }, + { + "type": "rich_text_preformatted", + "elements": [ + { + "type": "text", + "text": "disable_openlineage_plugin\ttrue\tDisable the OpenLineage plugin to avoid duplicative processing." + } + ], + "border": 0 + }, + { + "type": "rich_text_section", + "elements": [ + { + "type": "text", + "text": "They reuse the extractors but then “patch” the behavior." + } + ] + } + ] + } + ], + "team": "T01CWUYP5AR", + "attachments": [ + { + "image_url": "https://raw.githubusercontent.com/datahub-project/static-assets/main/imgs/airflow/plugin_connection_setup.png", + "image_width": 842, + "image_height": 643, + "image_bytes": 50334, + "id": 1, + "footer_icon": "https://slack.github.com/static/img/favicon-neutral.png", + "color": "24292f", + "bot_id": "B01VA0FB340", + "app_unfurl_url": "https://github.com/datahub-project/datahub/blob/2b0811b9875d7d7ea11fb01d0157a21fdd67f020/docs/lineage/airflow.md", + "is_app_unfurl": true, + "app_id": "A01BP7R4KNY", + "fallback": "842x643px image", + "text": "*Airflow Integration*\n\n:::note\n\nIf you're looking to schedule DataHub ingestion using Airflow, see the guide on <../../metadata-ingestion/schedule_docs/airflow.md|scheduling ingestion with Airflow>.\n\n:::\n\nThe DataHub Airflow plugin supports:\n\n• Automatic column-level lineage extraction from various operators e.g. `SqlOperator`s (including `MySqlOperator`, `PostgresOperator`, `SnowflakeOperator`, and more), `S3FileTransformOperator`, and a few others.\n• Airflow DAG and tasks, including properties, ownership, and tags.\n• Task run information, including task successes and failures.\n• Manual lineage annotations using `inlets` and `outlets` on Airflow operators.\n\nThere's two actively supported implementations of the plugin, with different Airflow version support.\n\nIf you're using Airflow older than 2.1, it's possible to use the v1 plugin with older versions of `acryl-datahub-airflow-plugin`. See the for more details.\n\n*DataHub Plugin v2*\n*Installation*\n\nThe v2 plugin requires Airflow 2.3+ and Python 3.8+. If you don't meet these requirements, use the v1 plugin instead.\n\n```\npip install 'acryl-datahub-airflow-plugin[plugin-v2]'\n```\n\n*Configuration*\n\nSet up a DataHub connection in Airflow, either via command line or the Airflow UI.\n\n*Command Line*\n\n```\nairflow connections add --conn-type 'datahub-rest' 'datahub_rest_default' --conn-host '' --conn-password ''\n```\n\n*Airflow UI*\n\nOn the Airflow UI, go to Admin -> Connections and click the \"+\" symbol to create a new connection. Select \"DataHub REST Server\" from the dropdown for \"Connection Type\" and enter the appropriate values.\n\n\n\n*Optional Configurations*\n\nNo additional configuration is required to use the plugin. However, there are some optional configuration parameters that can be set in the `airflow.cfg` file.\n\n```\n[datahub]\n# Optional - additional config here.\nenabled = True # default\n```\n\n*Automatic lineage extraction*\n\nTo automatically extract lineage information, the v2 plugin builds on top of Airflow's built-in .\n\nThe SQL-related extractors have been updated to use DataHub's SQL parser, which is more robust than the built-in one and uses DataHub's metadata information to generate column-level lineage. We discussed the DataHub SQL parser, including why schema-aware parsing works better and how it performs on benchmarks, during the .\n\n*DataHub Plugin v1*\n*Installation*\n\nThe v1 plugin requires Airflow 2.1+ and Python 3.8+. If you're on older versions, it's still possible to use an older version of the plugin. See the for more details.\n\nIf you're using Airflow 2.3+, we recommend using the v2 plugin instead. If you need to use the v1 plugin with Airflow 2.3+, you must also set the environment variable `DATAHUB_AIRFLOW_PLUGIN_USE_V1_PLUGIN=true`.\n\n```\npip install 'acryl-datahub-airflow-plugin[plugin-v1]'\n\n# The DataHub rest connection type is included by default.\n# To use the DataHub Kafka connection type, install the plugin with the kafka extras.\npip install 'acryl-datahub-airflow-plugin[plugin-v1,datahub-kafka]'\n```\n\n*Configuration*\n*Disable lazy plugin loading*\n\n```\n[core]\nlazy_load_plugins = False\n```\n\nOn MWAA you should add this config to your .\n\n*Setup a DataHub connection*\n\nYou must configure an Airflow connection for Datahub. We support both a Datahub REST and a Kafka-based connections, but you only need one.\n\n```\n# For REST-based:\nairflow connections add --conn-type 'datahub_rest' 'datahub_rest_default' --conn-host '' --conn-password ''\n# For Kafka-based (standard Kafka sink config can be passed via extras):\nairflow connections add --conn-type 'datahub_kafka' 'datahub_kafka_default' --conn-host 'broker:9092' --conn-extra '{}'\n```\n\n*Configure the plugin*\n\nIf your config doesn't align with the default values, you can configure the plugin in your `airflow.cfg` file.\n\n```\n[datahub]\nenabled = true\nconn_id = datahub_rest_default # or datahub_kafka_default\n# etc.\n```\n\n*Validate that the plugin is working*\n\n1. Go and check in Airflow at Admin -> Plugins menu if you can see the DataHub plugin\n2. Run an Airflow DAG. In the task logs, you should see Datahub related log messages like:\n\n```\nEmitting DataHub ...\n```\n\n*Manual Lineage Annotation*\n*Using `inlets` and `outlets`*\n\nYou can manually annotate lineage by setting `inlets` and `outlets` on your Airflow operators. This is useful if you're using an operator that doesn't support automatic lineage extraction, or if you want to override the automatic lineage extraction.\n\nWe have a few code samples that demonstrate how to use `inlets` and `outlets`:\n\n• <../../metadata-ingestion-modules/airflow-plugin/src/datahub_airflow_plugin/example_dags/lineage_backend_demo.py|`lineage_backend_demo.py`>\n• <../../metadata-ingestion-modules/airflow-plugin/src/datahub_airflow_plugin/example_dags/lineage_backend_taskflow_demo.py|`lineage_backend_taskflow_demo.py`> - uses the \n\nFor more information, take a look at the .\n\n*Custom Operators*\n\nIf you have created a that inherits from the BaseOperator class, \nwhen overriding the `execute` function, set inlets and outlets via `context['ti'].task.inlets` and `context['ti'].task.outlets`. \nThe DataHub Airflow plugin will then pick up those inlets and outlets after the task runs.\n\n```\nclass DbtOperator(BaseOperator):\n ...\n\n def execute(self, context):\n # do something\n inlets, outlets = self._get_lineage()\n # inlets/outlets are lists of either datahub_airflow_plugin.entities.Dataset or datahub_airflow_plugin.entities.Urn\n context['ti'].task.inlets = self.inlets\n context['ti'].task.outlets = self.outlets\n\n def _get_lineage(self):\n # Do some processing to get inlets/outlets\n\n return inlets, outlets\n```\n\nIf you override the `pre_execute` and `post_execute` function, ensure they include the `@prepare_lineage` and `@apply_lineage` decorators respectively. Reference the for more details.\n\n*Emit Lineage Directly*\n\nIf you can't use the plugin or annotate inlets/outlets, you can also emit lineage using the `DatahubEmitterOperator`.\n\nReference <../../metadata-ingestion-modules/airflow-plugin/src/datahub_airflow_plugin/example_dags/lineage_emission_dag.py|`lineage_emission_dag.py`> for a full example.\n\nIn order to use this example, you must first configure the Datahub hook. Like in ingestion, we support a Datahub REST hook and a Kafka-based hook. See the plugin configuration for examples.\n\n*Debugging*\n*Missing lineage*\n\nIf you're not seeing lineage in DataHub, check the following:\n\n• Validate that the plugin is loaded in Airflow. Go to Admin -> Plugins and check that the DataHub plugin is listed.\n• With the v2 plugin, it should also print a l…", + "title": "", + "footer": "", + "mrkdwn_in": [ + "text" + ] + } + ], + "thread_ts": "1700246539.228259", + "parent_user_id": "U01DCLP0GU9" + }, + { + "client_msg_id": "59a1e940-c1fc-490c-b9c5-147d5e834b71", + "type": "message", + "text": "Of course this approach will need changing again with AF 2.7", + "user": "U01DCLP0GU9", + "ts": "1700246932.436609", + "blocks": [ + { + "type": "rich_text", + "block_id": "/POFd", + "elements": [ + { + "type": "rich_text_section", + "elements": [ + { + "type": "text", + "text": "Of course this approach will need changing again with AF 2.7" + } + ] + } + ] + } + ], + "team": "T01CWUYP5AR", + "thread_ts": "1700246539.228259", + "parent_user_id": "U01DCLP0GU9" + }, + { + "client_msg_id": "8da9603b-c057-4ef0-8124-86412209192e", + "type": "message", + "text": "It’s their choice :shrug:", + "user": "U01DCLP0GU9", + "ts": "1700246942.225929", + "blocks": [ + { + "type": "rich_text", + "block_id": "XBzX4", + "elements": [ + { + "type": "rich_text_section", + "elements": [ + { + "type": "text", + "text": "It’s their choice " + }, + { + "type": "emoji", + "name": "shrug", + "unicode": "1f937" + } + ] + } + ] + } + ], + "team": "T01CWUYP5AR", + "thread_ts": "1700246539.228259", + "parent_user_id": "U01DCLP0GU9" + }, + { + "client_msg_id": "1447c2c4-00fc-4539-95c3-53e11be40529", + "type": "message", + "text": "It looks like we can possibly learn from their approach in SQL parsing: ", + "user": "U01DCLP0GU9", + "ts": "1700247083.921989", + "blocks": [ + { + "type": "rich_text", + "block_id": "GZG6c", + "elements": [ + { + "type": "rich_text_section", + "elements": [ + { + "type": "text", + "text": "It looks like we can possibly learn from their approach in SQL parsing: " + }, + { + "type": "link", + "url": "https://datahubproject.io/docs/lineage/airflow/#automatic-lineage-extraction" + } + ] + } + ] + } + ], + "team": "T01CWUYP5AR", + "attachments": [ + { + "from_url": "https://datahubproject.io/docs/lineage/airflow/#automatic-lineage-extraction", + "service_icon": "https://datahubproject.io/img/favicon.ico", + "id": 1, + "original_url": "https://datahubproject.io/docs/lineage/airflow/#automatic-lineage-extraction", + "fallback": "Airflow Integration | DataHub", + "text": "If you're looking to schedule DataHub ingestion using Airflow, see the guide on scheduling ingestion with Airflow.", + "title": "Airflow Integration | DataHub", + "title_link": "https://datahubproject.io/docs/lineage/airflow/#automatic-lineage-extraction", + "service_name": "datahubproject.io" + } + ], + "thread_ts": "1700246539.228259", + "parent_user_id": "U01DCLP0GU9" + }, + { + "client_msg_id": "17522ae0-7886-4487-a7a9-d8263e1700cf", + "type": "message", + "text": "what's that approach? I only know they have been claiming best SQL parsing capabilities", + "user": "U02S6F54MAB", + "ts": "1700257371.478449", + "blocks": [ + { + "type": "rich_text", + "block_id": "aj4Ne", + "elements": [ + { + "type": "rich_text_section", + "elements": [ + { + "type": "text", + "text": "what's that approach? I only know they have been claiming best SQL parsing capabilities" + } + ] + } + ] + } + ], + "team": "T01CWUYP5AR", + "thread_ts": "1700246539.228259", + "parent_user_id": "U01DCLP0GU9" + }, + { + "client_msg_id": "51a9f111-c022-4bbb-8948-d6a775d0ad9c", + "type": "message", + "text": "I haven’t looked in the details but I’m assuming it is in this repo. (my comment is entirely based on the claim here)", + "user": "U01DCLP0GU9", + "ts": "1700272488.934199", + "blocks": [ + { + "type": "rich_text", + "block_id": "rARhz", + "elements": [ + { + "type": "rich_text_section", + "elements": [ + { + "type": "text", + "text": "I haven’t looked in the details but I’m assuming it is in this repo. (my comment is entirely based on the claim here)" + } + ] + } + ] + } + ], + "team": "T01CWUYP5AR", + "edited": { + "user": "U01DCLP0GU9", + "ts": "1700272494.000000" + }, + "thread_ts": "1700246539.228259", + "parent_user_id": "U01DCLP0GU9" + }, + { + "client_msg_id": "90186ee8-27f8-4b0f-859a-0760106930b3", + "type": "message", + "text": "`` -> The interesting difference is that in order to find table schemas, they use their data catalog to evaluate column-level lineage instead of doing this on the client side.\n\nMy understanding by example is: If you do\n```create table x as select * from y```\nyou need to resolve `*` to know column level lineage. Our approach is to do that on the client side, probably with an extra call to database. Their approach is to do that based on the data catalog information.", + "user": "U02MK6YNAQ5", + "ts": "1700467087.549929", + "blocks": [ + { + "type": "rich_text", + "block_id": "j2CGA", + "elements": [ + { + "type": "rich_text_section", + "elements": [ + { + "type": "link", + "url": "https://www.acryldata.io/blog/extracting-column-level-lineage-from-sql", + "style": { + "code": true + } + }, + { + "type": "text", + "text": " -> The interesting difference is that in order to find table schemas, they use their data catalog to evaluate column-level lineage instead of doing this on the client side.\n\nMy understanding by example is: If you do\n" + } + ] + }, + { + "type": "rich_text_preformatted", + "elements": [ + { + "type": "text", + "text": "create table x as select * from y" + } + ], + "border": 0 + }, + { + "type": "rich_text_section", + "elements": [ + { + "type": "text", + "text": "you need to resolve " + }, + { + "type": "text", + "text": "*", + "style": { + "code": true + } + }, + { + "type": "text", + "text": " to know column level lineage. Our approach is to do that on the client side, probably with an extra call to database. Their approach is to do that based on the data catalog information." + } + ] + } + ] + } + ], + "team": "T01CWUYP5AR", + "thread_ts": "1700246539.228259", + "parent_user_id": "U01DCLP0GU9" + } + ] + }, + { + "client_msg_id": "290d0b95-e6bc-4529-b446-aa462954f331", + "type": "message", + "text": "CFP for Berlin Buzzwords went up: \nStill over 3 months to submit :slightly_smiling_face:", + "user": "U01RA9B5GG2", + "ts": "1700155042.082759", + "blocks": [ + { + "type": "rich_text", + "block_id": "IY1qh", + "elements": [ + { + "type": "rich_text_section", + "elements": [ + { + "type": "text", + "text": "CFP for Berlin Buzzwords went up: " + }, + { + "type": "link", + "url": "https://2024.berlinbuzzwords.de/call-for-papers/" + }, + { + "type": "text", + "text": "\nStill over 3 months to submit " + }, + { + "type": "emoji", + "name": "slightly_smiling_face", + "unicode": "1f642" + } + ] + } + ] + } + ], + "team": "T01CWUYP5AR", + "thread_ts": "1700155042.082759", + "reply_count": 3, + "reply_users_count": 2, + "latest_reply": "1700165993.306159", + "reply_users": [ + "U02LXF3HUN7", + "U02S6F54MAB" + ], + "is_locked": false, + "subscribed": true, + "last_read": "1700165993.306159", + "replies": [ + { + "client_msg_id": "59c2bed5-ed72-4bc1-aed1-c86f76e6baf6", + "type": "message", + "text": "thanks, updated the talks board", + "user": "U02LXF3HUN7", + "ts": "1700156576.787649", + "blocks": [ + { + "type": "rich_text", + "block_id": "uZOkT", + "elements": [ + { + "type": "rich_text_section", + "elements": [ + { + "type": "text", + "text": "thanks, updated the talks board" + } + ] + } + ] + } + ], + "team": "T01CWUYP5AR", + "thread_ts": "1700155042.082759", + "parent_user_id": "U01RA9B5GG2" + }, + { + "client_msg_id": "98d5c5e2-8039-4a6e-9833-9de669c9ab47", + "type": "message", + "text": "", + "user": "U02LXF3HUN7", + "ts": "1700156590.752459", + "blocks": [ + { + "type": "rich_text", + "block_id": "WwA6M", + "elements": [ + { + "type": "rich_text_section", + "elements": [ + { + "type": "link", + "url": "https://github.com/orgs/OpenLineage/projects/4/views/1" + } + ] + } + ] + } + ], + "team": "T01CWUYP5AR", + "thread_ts": "1700155042.082759", + "parent_user_id": "U01RA9B5GG2" + }, + { + "client_msg_id": "b5cd14ed-3246-4722-9385-24594e0b2d77", + "type": "message", + "text": "I'm in, will think what to talk about and appreciate any advice :slightly_smiling_face:", + "user": "U02S6F54MAB", + "ts": "1700165993.306159", + "blocks": [ + { + "type": "rich_text", + "block_id": "vK7e9", + "elements": [ + { + "type": "rich_text_section", + "elements": [ + { + "type": "text", + "text": "I'm in, will think what to talk about and appreciate any advice " + }, + { + "type": "emoji", + "name": "slightly_smiling_face", + "unicode": "1f642" + } + ] + } + ] + } + ], + "team": "T01CWUYP5AR", + "thread_ts": "1700155042.082759", + "parent_user_id": "U01RA9B5GG2" + } + ] + }, + { + "client_msg_id": "565052e6-10db-43e3-bf54-f56143b67702", + "type": "message", + "text": "worlds are colliding: 6point6 has been acquired by Accenture", + "user": "U02LXF3HUN7", + "ts": "1700145084.414099", + "blocks": [ + { + "type": "rich_text", + "block_id": "Phnmg", + "elements": [ + { + "type": "rich_text_section", + "elements": [ + { + "type": "text", + "text": "worlds are colliding: 6point6 has been acquired by Accenture" + } + ] + } + ] + } + ], + "team": "T01CWUYP5AR", + "thread_ts": "1700145084.414099", + "reply_count": 5, + "reply_users_count": 3, + "latest_reply": "1700151197.023279", + "reply_users": [ + "U02LXF3HUN7", + "U01RA9B5GG2", + "U01HNKK4XAM" + ], + "is_locked": false, + "subscribed": true, + "last_read": "1700151197.023279", + "replies": [ + { + "client_msg_id": "4e2e079f-ed27-46fc-8b0c-a160525f3d6a", + "type": "message", + "text": "", + "user": "U02LXF3HUN7", + "ts": "1700145119.237459", + "blocks": [ + { + "type": "rich_text", + "block_id": "p2OqU", + "elements": [ + { + "type": "rich_text_section", + "elements": [ + { + "type": "link", + "url": "https://newsroom.accenture.com/news/2023/accenture-to-expand-government-transformation-capabilities-in-the-uk-with-acquisition-of-6point6" + } + ] + } + ] + } + ], + "team": "T01CWUYP5AR", + "attachments": [ + { + "from_url": "https://newsroom.accenture.com/news/2023/accenture-to-expand-government-transformation-capabilities-in-the-uk-with-acquisition-of-6point6", + "image_url": "https://newsroom.accenture.com/news/2023/media_1a9921811b72c2ade3a6d22550b92f42d782dcd12.jpeg?width=1200&format=pjpg&optimize=medium", + "image_width": 1200, + "image_height": 628, + "image_bytes": 60944, + "service_icon": "https://newsroom.accenture.com/favicon.ico", + "id": 1, + "original_url": "https://newsroom.accenture.com/news/2023/accenture-to-expand-government-transformation-capabilities-in-the-uk-with-acquisition-of-6point6", + "fallback": "Accenture to Expand Government Transformation Capabilities in the U.K. with Acquisition of 6point6", + "text": "Accenture has signed an agreement to acquire 6point6, a U.K. technology consultancy, specializing in cloud, data, and cybersecurity.", + "title": "Accenture to Expand Government Transformation Capabilities in the U.K. with Acquisition of 6point6", + "title_link": "https://newsroom.accenture.com/news/2023/accenture-to-expand-government-transformation-capabilities-in-the-uk-with-acquisition-of-6point6", + "service_name": "newsroom.accenture.com" + } + ], + "thread_ts": "1700145084.414099", + "parent_user_id": "U02LXF3HUN7" + }, + { + "client_msg_id": "5c7159ef-848b-44ef-b725-c7ffdb68e7a6", + "type": "message", + "text": "We should sell OL to governments", + "user": "U01RA9B5GG2", + "ts": "1700147007.152929", + "blocks": [ + { + "type": "rich_text", + "block_id": "SxApD", + "elements": [ + { + "type": "rich_text_section", + "elements": [ + { + "type": "text", + "text": "We should sell OL to governments" + } + ] + } + ] + } + ], + "team": "T01CWUYP5AR", + "thread_ts": "1700145084.414099", + "parent_user_id": "U02LXF3HUN7", + "reactions": [ + { + "name": "upside_down_face", + "users": [ + "U01HNKK4XAM" + ], + "count": 1 + } + ] + }, + { + "client_msg_id": "3b80d5c4-c44c-42b5-a8be-e29cb54d6524", + "type": "message", + "text": "we may have to rebrand to ClosedLineage", + "user": "U01HNKK4XAM", + "ts": "1700148036.394269", + "blocks": [ + { + "type": "rich_text", + "block_id": "Jxqqu", + "elements": [ + { + "type": "rich_text_section", + "elements": [ + { + "type": "text", + "text": "we may have to rebrand to ClosedLineage" + } + ] + } + ] + } + ], + "team": "T01CWUYP5AR", + "thread_ts": "1700145084.414099", + "parent_user_id": "U02LXF3HUN7" + }, + { + "client_msg_id": "ccf1d225-5ba1-4385-bde0-9e8170f77e0d", + "type": "message", + "text": "not in this way; just emit any event second time to secret NSA endpoint", + "user": "U01RA9B5GG2", + "ts": "1700148217.698289", + "blocks": [ + { + "type": "rich_text", + "block_id": "lCJt9", + "elements": [ + { + "type": "rich_text_section", + "elements": [ + { + "type": "text", + "text": "not in this way; just emit any event second time to secret NSA endpoint" + } + ] + } + ] + } + ], + "team": "T01CWUYP5AR", + "thread_ts": "1700145084.414099", + "parent_user_id": "U02LXF3HUN7" + }, + { + "client_msg_id": "d64e9f16-21eb-4907-a8b2-cce44cdcb728", + "type": "message", + "text": "we would need to improve our stock photo game", + "user": "U02LXF3HUN7", + "ts": "1700151197.023279", + "blocks": [ + { + "type": "rich_text", + "block_id": "HxE/r", + "elements": [ + { + "type": "rich_text_section", + "elements": [ + { + "type": "text", + "text": "we would need to improve our stock photo game" + } + ] + } + ] + } + ], + "team": "T01CWUYP5AR", + "thread_ts": "1700145084.414099", + "parent_user_id": "U02LXF3HUN7" + } + ] + }, + { + "client_msg_id": "79a4d1c5-4281-413d-910f-3da8d91cb07f", + "type": "message", + "text": "Any opinions about a free task management alternative to the free version of Notion (10-person limit)? Looking at Trello for keeping track of talks.", + "user": "U02LXF3HUN7", + "ts": "1700088623.669029", + "blocks": [ + { + "type": "rich_text", + "block_id": "9mtSc", + "elements": [ + { + "type": "rich_text_section", + "elements": [ + { + "type": "text", + "text": "Any opinions about a free task management alternative to the free version of Notion (10-person limit)? Looking at Trello for keeping track of talks." + } + ] + } + ] + } + ], + "team": "T01CWUYP5AR", + "thread_ts": "1700088623.669029", + "reply_count": 3, + "reply_users_count": 2, + "latest_reply": "1700148214.523069", + "reply_users": [ + "U01HNKK4XAM", + "U02LXF3HUN7" + ], + "is_locked": false, + "subscribed": true, + "last_read": "1700148214.523069", + "replies": [ + { + "client_msg_id": "9815972B-96D0-41B8-B384-BA12F3973205", + "type": "message", + "text": "What about GitHub projects?", + "user": "U01HNKK4XAM", + "ts": "1700094737.005179", + "blocks": [ + { + "type": "rich_text", + "block_id": "QcAzj", + "elements": [ + { + "type": "rich_text_section", + "elements": [ + { + "type": "text", + "text": "What about GitHub projects?" + } + ] + } + ] + } + ], + "team": "T01CWUYP5AR", + "thread_ts": "1700088623.669029", + "parent_user_id": "U02LXF3HUN7", + "reactions": [ + { + "name": "+1", + "users": [ + "U02LXF3HUN7" + ], + "count": 1 + } + ] + }, + { + "client_msg_id": "a49931fb-de30-4a1d-868f-c73cfe634843", + "type": "message", + "text": "Projects is the way to go, thanks", + "user": "U02LXF3HUN7", + "ts": "1700144866.061859", + "blocks": [ + { + "type": "rich_text", + "block_id": "w9wSZ", + "elements": [ + { + "type": "rich_text_section", + "elements": [ + { + "type": "text", + "text": "Projects is the way to go, thanks" + } + ] + } + ] + } + ], + "team": "T01CWUYP5AR", + "thread_ts": "1700088623.669029", + "parent_user_id": "U02LXF3HUN7" + }, + { + "client_msg_id": "a1ce4508-002c-4ff3-91e8-27b9b2c898d0", + "type": "message", + "text": "Set up a Projects board. New projects are private by default. We could make it public. The one thing that’s missing that we could use is a built-in date field for alerting about upcoming deadlines…", + "user": "U02LXF3HUN7", + "ts": "1700148214.523069", + "blocks": [ + { + "type": "rich_text", + "block_id": "eEnTu", + "elements": [ + { + "type": "rich_text_section", + "elements": [ + { + "type": "text", + "text": "Set up a Projects board. New projects are private by default. We could make it public. The one thing that’s missing that we could use is a built-in date field for alerting about upcoming deadlines…" + } + ] + } + ] + } + ], + "team": "T01CWUYP5AR", + "thread_ts": "1700088623.669029", + "parent_user_id": "U02LXF3HUN7", + "reactions": [ + { + "name": "raised_hands", + "users": [ + "U01HNKK4XAM", + "U01RA9B5GG2" + ], + "count": 2 + } + ] + } + ] + }, + { + "client_msg_id": "56357d69-1804-4184-8e84-c307a173b4d7", + "type": "message", + "text": "have we discussed adding column level lineage support to Airflow? ", + "user": "U01DCMDFHBK", + "ts": "1700087546.032789", + "blocks": [ + { + "type": "rich_text", + "block_id": "/WCoQ", + "elements": [ + { + "type": "rich_text_section", + "elements": [ + { + "type": "text", + "text": "have we discussed adding column level lineage support to Airflow? " + }, + { + "type": "link", + "url": "https://marquezproject.slack.com/archives/C01E8MQGJP7/p1700087438599279?thread_ts=1700084629.245949&cid=C01E8MQGJP7" + } + ] + } + ] + } + ], + "team": "T01CWUYP5AR", + "thread_ts": "1700087546.032789", + "reply_count": 3, + "reply_users_count": 2, + "latest_reply": "1700087719.745889", + "reply_users": [ + "U02S6F54MAB", + "U01DCMDFHBK" + ], + "is_locked": false, + "subscribed": false, + "replies": [ + { + "client_msg_id": "b6c39a4c-39b1-4580-be60-d8a545e0e4c0", + "type": "message", + "text": "we have it in SQL operators", + "user": "U02S6F54MAB", + "ts": "1700087599.744449", + "blocks": [ + { + "type": "rich_text", + "block_id": "UED2i", + "elements": [ + { + "type": "rich_text_section", + "elements": [ + { + "type": "text", + "text": "we have it in SQL operators" + } + ] + } + ] + } + ], + "team": "T01CWUYP5AR", + "thread_ts": "1700087546.032789", + "parent_user_id": "U01DCMDFHBK" + }, + { + "client_msg_id": "e94cf9ac-c1f8-4174-afc4-c9e721d069ae", + "type": "message", + "text": "OOh any docs / code? or if you’d like to respond in the MQZ slack :pray:", + "user": "U01DCMDFHBK", + "ts": "1700087665.101029", + "blocks": [ + { + "type": "rich_text", + "block_id": "xEi1G", + "elements": [ + { + "type": "rich_text_section", + "elements": [ + { + "type": "text", + "text": "OOh any docs / code? or if you’d like to respond in the MQZ slack " + }, + { + "type": "emoji", + "name": "pray", + "unicode": "1f64f" + } + ] + } + ] + } + ], + "team": "T01CWUYP5AR", + "thread_ts": "1700087546.032789", + "parent_user_id": "U01DCMDFHBK" + }, + { + "client_msg_id": "d8406883-16b7-4722-a5a5-c4865a9d2cea", + "type": "message", + "text": "I’ll reply there", + "user": "U02S6F54MAB", + "ts": "1700087719.745889", + "blocks": [ + { + "type": "rich_text", + "block_id": "lm1RD", + "elements": [ + { + "type": "rich_text_section", + "elements": [ + { + "type": "text", + "text": "I’ll reply there" + } + ] + } + ] + } + ], + "team": "T01CWUYP5AR", + "thread_ts": "1700087546.032789", + "parent_user_id": "U01DCMDFHBK", + "reactions": [ + { + "name": "heart", + "users": [ + "U01DCMDFHBK", + "U01HNKK4XAM" + ], + "count": 2 + } + ] + } + ] + }, + { + "client_msg_id": "1c768cec-a0dc-49c7-b912-53c6f5b2674b", + "type": "message", + "text": "Apparently an admin can view a Slack archive at any time at this URL: . Only public channels are available, though.", + "user": "U02LXF3HUN7", + "ts": "1700078359.877599", + "blocks": [ + { + "type": "rich_text", + "block_id": "kIAVN", + "elements": [ + { + "type": "rich_text_section", + "elements": [ + { + "type": "text", + "text": "Apparently an admin can view a Slack archive at any time at this URL: " + }, + { + "type": "link", + "url": "https://openlineage.slack.com/services/export", + "text": "https://openlineage.slack.com/services/export" + }, + { + "type": "text", + "text": ". Only public channels are available, though." + } + ] + } + ] + } + ], + "team": "T01CWUYP5AR", + "thread_ts": "1700078359.877599", + "reply_count": 1, + "reply_users_count": 1, + "latest_reply": "1700085189.085139", + "reply_users": [ + "U01DCLP0GU9" + ], + "is_locked": false, + "subscribed": true, + "last_read": "1700085189.085139", + "replies": [ + { + "client_msg_id": "345bfb5e-683e-461a-88ee-49e319b9bab1", + "type": "message", + "text": "you are now admin", + "user": "U01DCLP0GU9", + "ts": "1700085189.085139", + "blocks": [ + { + "type": "rich_text", + "block_id": "UEiAC", + "elements": [ + { + "type": "rich_text_section", + "elements": [ + { + "type": "text", + "text": "you are now admin" + } + ] + } + ] + } + ], + "team": "T01CWUYP5AR", + "thread_ts": "1700078359.877599", + "parent_user_id": "U02LXF3HUN7", + "reactions": [ + { + "name": "+1", + "users": [ + "U02LXF3HUN7" + ], + "count": 1 + } + ] + } + ] + }, + { + "client_msg_id": "cd28f84f-6bdd-4095-9149-2979383a7e9c", + "type": "message", + "text": "Anyone have thoughts about how to address the question about “pain points” here? . (Listing pros is easy — it’s the cons we don’t have boilerplate for)", + "user": "U02LXF3HUN7", + "ts": "1700078230.775579", + "blocks": [ + { + "type": "rich_text", + "block_id": "MocUx", + "elements": [ + { + "type": "rich_text_section", + "elements": [ + { + "type": "text", + "text": "Anyone have thoughts about how to address the question about “pain points” here? " + }, + { + "type": "link", + "url": "https://openlineage.slack.com/archives/C01CK9T7HKR/p1700064564825909", + "text": "https://openlineage.slack.com/archives/C01CK9T7HKR/p1700064564825909" + }, + { + "type": "text", + "text": ". (Listing pros is easy — it’s the cons we don’t have boilerplate for)" + } + ] + } + ] + } + ], + "team": "T01CWUYP5AR", + "attachments": [ + { + "from_url": "https://openlineage.slack.com/archives/C01CK9T7HKR/p1700064564825909", + "ts": "1700064564.825909", + "author_id": "U066HKFCHUG", + "channel_id": "C01CK9T7HKR", + "channel_team": "T01CWUYP5AR", + "is_msg_unfurl": true, + "message_blocks": [ + { + "team": "T01CWUYP5AR", + "channel": "C01CK9T7HKR", + "ts": "1700064564.825909", + "message": { + "blocks": [ + { + "type": "rich_text", + "block_id": "V52kz", + "elements": [ + { + "type": "rich_text_section", + "elements": [ + { + "type": "text", + "text": "Can anyone tell me why OL is better than other competitors if you can provide an analysis that would be great" + } + ] + } + ] + } + ] + } + } + ], + "id": 1, + "original_url": "https://openlineage.slack.com/archives/C01CK9T7HKR/p1700064564825909", + "fallback": "[November 15th, 2023 8:09 AM] naresh.naresh36: Can anyone tell me why OL is better than other competitors if you can provide an analysis that would be great", + "text": "Can anyone tell me why OL is better than other competitors if you can provide an analysis that would be great", + "author_name": "Naresh reddy", + "author_link": "https://openlineage.slack.com/team/U066HKFCHUG", + "author_icon": "https://avatars.slack-edge.com/2023-11-15/6192035069510_91e664fce7c3faeee1e7_48.jpg", + "author_subname": "Naresh reddy", + "mrkdwn_in": [ + "text" + ], + "footer": "Slack Conversation" + } + ], + "thread_ts": "1700078230.775579", + "reply_count": 2, + "reply_users_count": 2, + "latest_reply": "1700082291.373659", + "reply_users": [ + "U02LXF3HUN7", + "U01RA9B5GG2" + ], + "is_locked": false, + "subscribed": true, + "last_read": "1700082291.373659", + "replies": [ + { + "client_msg_id": "31c986c9-ecb0-4f3d-b0ef-11bc56dc0ea5", + "type": "message", + "text": "Maybe something like “OL has many desirable integrations, including a best-in-class Spark integration, but it’s like any other open standard in that it requires contributions in order to approach total coverage. Thankfully, we have many active contributors, and integrations are being added or improved upon all the time.”", + "user": "U02LXF3HUN7", + "ts": "1700078288.125819", + "blocks": [ + { + "type": "rich_text", + "block_id": "0bx39", + "elements": [ + { + "type": "rich_text_section", + "elements": [ + { + "type": "text", + "text": "Maybe something like “OL has many desirable integrations, including a best-in-class Spark integration, but it’s like any other open standard in that it requires contributions in order to approach total coverage. Thankfully, we have many active contributors, and integrations are being added or improved upon all the time.”" + } + ] + } + ] + } + ], + "team": "T01CWUYP5AR", + "thread_ts": "1700078230.775579", + "parent_user_id": "U02LXF3HUN7" + }, + { + "client_msg_id": "16f88fa1-980e-4cac-bba8-fd2592cc3cf2", + "type": "message", + "text": "Maybe rephrase pain points to \"something we're not actively focusing on\"", + "user": "U01RA9B5GG2", + "ts": "1700082291.373659", + "blocks": [ + { + "type": "rich_text", + "block_id": "EqOgW", + "elements": [ + { + "type": "rich_text_section", + "elements": [ + { + "type": "text", + "text": "Maybe rephrase pain points to \"something we're not actively focusing on\"" + } + ] + } + ] + } + ], + "team": "T01CWUYP5AR", + "thread_ts": "1700078230.775579", + "parent_user_id": "U02LXF3HUN7" + } + ] + }, + { + "type": "message", + "text": "is it time to *support hudi*?", + "files": [ + { + "id": "F065PUT9SRL", + "created": 1700068641, + "timestamp": 1700068641, + "name": "image.png", + "title": "image.png", + "mimetype": "image/png", + "filetype": "png", + "pretty_type": "PNG", + "user": "U02S6F54MAB", + "user_team": "T01CWUYP5AR", + "editable": false, + "size": 644916, + "mode": "hosted", + "is_external": false, + "external_type": "", + "is_public": true, + "public_url_shared": false, + "display_as_bot": false, + "username": "", + "url_private": "https://files.slack.com/files-pri/T01CWUYP5AR-F065PUT9SRL/image.png", + "url_private_download": "https://files.slack.com/files-pri/T01CWUYP5AR-F065PUT9SRL/download/image.png", + "media_display_type": "unknown", + "thumb_64": "https://files.slack.com/files-tmb/T01CWUYP5AR-F065PUT9SRL-cf00de4cfd/image_64.png", + "thumb_80": "https://files.slack.com/files-tmb/T01CWUYP5AR-F065PUT9SRL-cf00de4cfd/image_80.png", + "thumb_360": "https://files.slack.com/files-tmb/T01CWUYP5AR-F065PUT9SRL-cf00de4cfd/image_360.png", + "thumb_360_w": 360, + "thumb_360_h": 300, + "thumb_480": "https://files.slack.com/files-tmb/T01CWUYP5AR-F065PUT9SRL-cf00de4cfd/image_480.png", + "thumb_480_w": 480, + "thumb_480_h": 400, + "thumb_160": "https://files.slack.com/files-tmb/T01CWUYP5AR-F065PUT9SRL-cf00de4cfd/image_160.png", + "thumb_720": "https://files.slack.com/files-tmb/T01CWUYP5AR-F065PUT9SRL-cf00de4cfd/image_720.png", + "thumb_720_w": 720, + "thumb_720_h": 600, + "thumb_800": "https://files.slack.com/files-tmb/T01CWUYP5AR-F065PUT9SRL-cf00de4cfd/image_800.png", + "thumb_800_w": 800, + "thumb_800_h": 666, + "thumb_960": "https://files.slack.com/files-tmb/T01CWUYP5AR-F065PUT9SRL-cf00de4cfd/image_960.png", + "thumb_960_w": 960, + "thumb_960_h": 800, + "thumb_1024": "https://files.slack.com/files-tmb/T01CWUYP5AR-F065PUT9SRL-cf00de4cfd/image_1024.png", + "thumb_1024_w": 1024, + "thumb_1024_h": 853, + "original_w": 1174, + "original_h": 978, + "thumb_tiny": "AwAnADCmZy4QEKNi4FNZhjn9KbEQC2e444p+Pb9KAI8j1P5Uny+p/KpDnsufwpvz+h/KgBvy+p/Kg47E/lTvn9D+VNKt1IP5UAXNKOLlieyH+laYbOcSk/h0rM0r/j6P+4f6VrjdQAb19aaZkBwTj8Kd81KM0xCB1IyDxTLkg2suOfkP8qJSQRzio5CTbT5OfkP8qLBfWxm6Y225PqVOK1xJxzjP41jaf/x9D6GtWkMlMnPGMUCQd8VFRQA5XLKC6qG/Om3DqLaXoMqegoqG6/49pP8AdoA//9k=", + "permalink": "https://openlineage.slack.com/files/U02S6F54MAB/F065PUT9SRL/image.png", + "permalink_public": "https://slack-files.com/T01CWUYP5AR-F065PUT9SRL-56a680662b", + "is_starred": false, + "has_rich_preview": false, + "file_access": "visible" + } + ], + "upload": false, + "user": "U02S6F54MAB", + "display_as_bot": false, + "ts": "1700068651.517579", + "blocks": [ + { + "type": "rich_text", + "block_id": "bDAOH", + "elements": [ + { + "type": "rich_text_section", + "elements": [ + { + "type": "text", + "text": "is it time to " + }, + { + "type": "text", + "text": "support hudi", + "style": { + "bold": true + } + }, + { + "type": "text", + "text": "?" + } + ] + } + ] + } + ], + "client_msg_id": "d5da62dd-e797-486b-9aae-a5acdbe4a1c8", + "reactions": [ + { + "name": "joy", + "users": [ + "U01HNKK4XAM" + ], + "count": 1 + } + ] + }, + { + "client_msg_id": "fdc57b50-c9cf-4b13-be38-681a0a2cbfeb", + "type": "message", + "text": "Got the doc + poc for hook-level coverage: ", + "user": "U01RA9B5GG2", + "ts": "1700066684.350639", + "blocks": [ + { + "type": "rich_text", + "block_id": "gkEfy", + "elements": [ + { + "type": "rich_text_section", + "elements": [ + { + "type": "text", + "text": "Got the doc + poc for hook-level coverage: " + }, + { + "type": "link", + "url": "https://docs.google.com/document/d/1q0shiUxopASO8glgMqjDn89xigJnGrQuBMbcRdolUdk/edit?usp=sharing" + } + ] + } + ] + } + ], + "team": "T01CWUYP5AR", + "thread_ts": "1700066684.350639", + "reply_count": 2, + "reply_users_count": 2, + "latest_reply": "1700069197.574369", + "reply_users": [ + "U02S6F54MAB", + "U01RA9B5GG2" + ], + "is_locked": false, + "subscribed": false, + "reactions": [ + { + "name": "eyes", + "users": [ + "U02S6F54MAB" + ], + "count": 1 + } + ], + "replies": [ + { + "client_msg_id": "d1c376d9-efa7-4a7e-9d43-4b7864c43c99", + "type": "message", + "text": "did you check if `LineageCollector` is instantiated once per process?", + "user": "U02S6F54MAB", + "ts": "1700069067.090359", + "blocks": [ + { + "type": "rich_text", + "block_id": "AbONi", + "elements": [ + { + "type": "rich_text_section", + "elements": [ + { + "type": "text", + "text": "did you check if " + }, + { + "type": "text", + "text": "LineageCollector", + "style": { + "code": true + } + }, + { + "type": "text", + "text": " is instantiated once per process?" + } + ] + } + ] + } + ], + "team": "T01CWUYP5AR", + "edited": { + "user": "U02S6F54MAB", + "ts": "1700069074.000000" + }, + "thread_ts": "1700066684.350639", + "parent_user_id": "U01RA9B5GG2" + }, + { + "client_msg_id": "b0881b41-f5e0-43de-83ce-0c6b28480e59", + "type": "message", + "text": "Using it only via `get_hook_lineage_collector`", + "user": "U01RA9B5GG2", + "ts": "1700069197.574369", + "blocks": [ + { + "type": "rich_text", + "block_id": "2tsBk", + "elements": [ + { + "type": "rich_text_section", + "elements": [ + { + "type": "text", + "text": "Using it only via " + }, + { + "type": "text", + "text": "get_hook_lineage_collector", + "style": { + "code": true + } + } + ] + } + ] + } + ], + "team": "T01CWUYP5AR", + "thread_ts": "1700066684.350639", + "parent_user_id": "U01RA9B5GG2" + } + ] + }, + { + "type": "message", + "subtype": "thread_broadcast", + "text": "hey look, more fun\n", + "user": "U02S6F54MAB", + "ts": "1700040937.040239", + "thread_ts": "1700004648.584649", + "root": { + "client_msg_id": "e27ec408-535d-4a9b-a41b-c848a8cb286a", + "type": "message", + "text": "\nfun PR incoming", + "user": "U02S6F54MAB", + "ts": "1700004648.584649", + "blocks": [ + { + "type": "rich_text", + "block_id": "xxBfv", + "elements": [ + { + "type": "rich_text_section", + "elements": [ + { + "type": "link", + "url": "https://github.com/OpenLineage/OpenLineage/pull/2260" + }, + { + "type": "text", + "text": "\nfun PR incoming" + } + ] + } + ] + } + ], + "team": "T01CWUYP5AR", + "thread_ts": "1700004648.584649", + "reply_count": 5, + "reply_users_count": 4, + "latest_reply": "1700066753.852469", + "reply_users": [ + "U02S6F54MAB", + "U01RA9B5GG2", + "U02MK6YNAQ5", + "U01HNKK4XAM" + ], + "is_locked": false, + "subscribed": false + }, + "blocks": [ + { + "type": "rich_text", + "block_id": "yGPSQ", + "elements": [ + { + "type": "rich_text_section", + "elements": [ + { + "type": "text", + "text": "hey look, more fun\n" + }, + { + "type": "link", + "url": "https://github.com/OpenLineage/OpenLineage/pull/2263" + } + ] + } + ] + } + ], + "client_msg_id": "9fbbabe3-ab1f-4f16-8e55-28f574dd7eb5" + }, + { + "client_msg_id": "12cbe1a1-4a86-4be0-9725-2e7a0b053ea0", + "type": "message", + "text": "also, what about this PR? ", + "user": "U01DCMDFHBK", + "ts": "1700037370.235629", + "blocks": [ + { + "type": "rich_text", + "block_id": "8bSTa", + "elements": [ + { + "type": "rich_text_section", + "elements": [ + { + "type": "text", + "text": "also, what about this PR? " + }, + { + "type": "link", + "url": "https://github.com/MarquezProject/marquez/pull/2654" + } + ] + } + ] + } + ], + "team": "T01CWUYP5AR", + "attachments": [ + { + "id": 1, + "footer_icon": "https://slack.github.com/static/img/favicon-neutral.png", + "ts": 1697557750, + "color": "36a64f", + "bot_id": "B01VA0FB340", + "app_unfurl_url": "https://github.com/MarquezProject/marquez/pull/2654", + "is_app_unfurl": true, + "app_id": "A01BP7R4KNY", + "fallback": "#2654 Runless events - refactor job_versions_io_mapping", + "text": "*Problem*\n\nThis is currently a draft PR which is far from being merged. It is missing few tests related to schema changes which are marked with `todo` within the code. I've created a PR to have a better discussion on adding `job_id` to `job_versions_io_mapping`. This PR should be a follow-up of .\n\nThe assumption was that it should be helpful in optimising get-lineage query. I would like first to clarify how are we going to make benefit of this extra column.\n\n*Solution*\n\nPlease describe your change as it relates to the problem, or bug fix, as well as any dependencies. If your change requires a database schema migration, please describe the schema modification(s) and whether it's a _backwards-incompatible_ or _backwards-compatible_ change.\n\n> *Note:* All database schema changes require discussion. Please for context.\n\nOne-line summary:\n\n*Checklist*\n\n☐ You've your work\n☐ Your changes are accompanied by tests (_if relevant_)\n☐ Your change contains a and is self-contained\n☐ You've updated any relevant documentation (_if relevant_)\n☐ You've included a one-line summary of your change for the (_Depending on the change, this may not be necessary_).\n☐ You've versioned your `.sql` database schema migration according to (_if relevant_)\n☐ You've included a in any source code files (_if relevant_)", + "title": "#2654 Runless events - refactor job_versions_io_mapping", + "title_link": "https://github.com/MarquezProject/marquez/pull/2654", + "footer": "", + "fields": [ + { + "value": "docs, api", + "title": "Labels", + "short": true + }, + { + "value": "4", + "title": "Comments", + "short": true + } + ], + "mrkdwn_in": [ + "text" + ] + } + ], + "thread_ts": "1700037370.235629", + "reply_count": 4, + "reply_users_count": 1, + "latest_reply": "1700037512.239479", + "reply_users": [ + "U02MK6YNAQ5" + ], + "is_locked": false, + "subscribed": false, + "replies": [ + { + "client_msg_id": "693813d2-d928-4d99-bd74-95c0261714d4", + "type": "message", + "text": "this is the next to go", + "user": "U02MK6YNAQ5", + "ts": "1700037393.838709", + "blocks": [ + { + "type": "rich_text", + "block_id": "8GVNn", + "elements": [ + { + "type": "rich_text_section", + "elements": [ + { + "type": "text", + "text": "this is the next to go" + } + ] + } + ] + } + ], + "team": "T01CWUYP5AR", + "thread_ts": "1700037370.235629", + "parent_user_id": "U01DCMDFHBK" + }, + { + "client_msg_id": "cc02b08a-7cec-487a-a5d1-2cbad675b35e", + "type": "message", + "text": "and i consider it ready", + "user": "U02MK6YNAQ5", + "ts": "1700037398.104819", + "blocks": [ + { + "type": "rich_text", + "block_id": "lxJ9f", + "elements": [ + { + "type": "rich_text_section", + "elements": [ + { + "type": "text", + "text": "and i consider it ready" + } + ] + } + ] + } + ], + "team": "T01CWUYP5AR", + "thread_ts": "1700037370.235629", + "parent_user_id": "U01DCMDFHBK" + }, + { + "client_msg_id": "ce36fb29-57bb-475a-b7da-4f979ab4c96f", + "type": "message", + "text": "Then we have a draft one with streaming support -> which has an integration test of lineage endpoint working for streaming jobs", + "user": "U02MK6YNAQ5", + "ts": "1700037451.758639", + "blocks": [ + { + "type": "rich_text", + "block_id": "sO/AE", + "elements": [ + { + "type": "rich_text_section", + "elements": [ + { + "type": "text", + "text": "Then we have a draft one with streaming support " + }, + { + "type": "link", + "url": "https://github.com/MarquezProject/marquez/pull/2682/files" + }, + { + "type": "text", + "text": " -> which has an integration test of lineage endpoint working for streaming jobs" + } + ] + } + ] + } + ], + "team": "T01CWUYP5AR", + "thread_ts": "1700037370.235629", + "parent_user_id": "U01DCMDFHBK" + }, + { + "client_msg_id": "192e703d-06e9-4072-a252-9fd7d8cbdace", + "type": "message", + "text": "I still need to work on #2682 but you can review #2654. once you get some sleep, of course :wink:", + "user": "U02MK6YNAQ5", + "ts": "1700037512.239479", + "blocks": [ + { + "type": "rich_text", + "block_id": "jWaV5", + "elements": [ + { + "type": "rich_text_section", + "elements": [ + { + "type": "text", + "text": "I still need to work on #2682 but you can review #2654. once you get some sleep, of course " + }, + { + "type": "emoji", + "name": "wink", + "unicode": "1f609" + } + ] + } + ] + } + ], + "team": "T01CWUYP5AR", + "thread_ts": "1700037370.235629", + "parent_user_id": "U01DCMDFHBK", + "reactions": [ + { + "name": "heart", + "users": [ + "U01DCMDFHBK" + ], + "count": 1 + } + ] + } + ] + }, + { + "client_msg_id": "00f5d485-d95e-4616-9c02-57c0fceaf671", + "type": "message", + "text": "`_Minor_: We can consider defining a _run_state column and eventually dropping the event_type. That is, we can consider columns prefixed with _ to be \"remappings\" of OL properties to Marquez.` -> didn't get this one. Is it for now or some future plans?", + "user": "U02MK6YNAQ5", + "ts": "1700037282.474539", + "blocks": [ + { + "type": "rich_text", + "block_id": "iPdJ9", + "elements": [ + { + "type": "rich_text_section", + "elements": [ + { + "type": "text", + "text": "Minor", + "style": { + "italic": true, + "code": true + } + }, + { + "type": "text", + "text": ": We can consider defining a _run_state column and eventually dropping the event_type. That is, we can consider columns prefixed with _ to be \"remappings\" of OL properties to Marquez.", + "style": { + "code": true + } + }, + { + "type": "text", + "text": " -> didn't get this one. Is it for now or some future plans?" + } + ] + } + ] + } + ], + "team": "T01CWUYP5AR", + "thread_ts": "1700037282.474539", + "reply_count": 3, + "reply_users_count": 2, + "latest_reply": "1700037383.673919", + "reply_users": [ + "U01DCMDFHBK", + "U02MK6YNAQ5" + ], + "is_locked": false, + "subscribed": false, + "replies": [ + { + "client_msg_id": "8a385af8-864b-44d0-b0d7-92b750f37d59", + "type": "message", + "text": "future :wink:", + "user": "U01DCMDFHBK", + "ts": "1700037362.060349", + "blocks": [ + { + "type": "rich_text", + "block_id": "te6E7", + "elements": [ + { + "type": "rich_text_section", + "elements": [ + { + "type": "text", + "text": "future " + }, + { + "type": "emoji", + "name": "wink", + "unicode": "1f609" + } + ] + } + ] + } + ], + "team": "T01CWUYP5AR", + "thread_ts": "1700037282.474539", + "parent_user_id": "U02MK6YNAQ5" + }, + { + "client_msg_id": "53f83f97-23da-45cf-81ae-2282be94ce49", + "type": "message", + "text": "ok", + "user": "U02MK6YNAQ5", + "ts": "1700037370.551639", + "blocks": [ + { + "type": "rich_text", + "block_id": "luWBi", + "elements": [ + { + "type": "rich_text_section", + "elements": [ + { + "type": "text", + "text": "ok" + } + ] + } + ] + } + ], + "team": "T01CWUYP5AR", + "thread_ts": "1700037282.474539", + "parent_user_id": "U02MK6YNAQ5" + }, + { + "client_msg_id": "602b1385-218e-4076-be7b-f2b03a428b17", + "type": "message", + "text": "I will then replace enum with string", + "user": "U02MK6YNAQ5", + "ts": "1700037383.673919", + "blocks": [ + { + "type": "rich_text", + "block_id": "xUGoP", + "elements": [ + { + "type": "rich_text_section", + "elements": [ + { + "type": "text", + "text": "I will then replace enum with string" + } + ] + } + ] + } + ], + "team": "T01CWUYP5AR", + "thread_ts": "1700037282.474539", + "parent_user_id": "U02MK6YNAQ5" + } + ] + }, + { + "client_msg_id": "5d39e88b-3350-4baa-9474-38a624950aa7", + "type": "message", + "text": "<@U02MK6YNAQ5> approved PR with minor comments, I think the is one comment we’ll need to address before merging; otherwise solid work dude :ok_hand:", + "user": "U01DCMDFHBK", + "ts": "1700037147.106879", + "blocks": [ + { + "type": "rich_text", + "block_id": "ktmE8", + "elements": [ + { + "type": "rich_text_section", + "elements": [ + { + "type": "user", + "user_id": "U02MK6YNAQ5" + }, + { + "type": "text", + "text": " approved PR " + }, + { + "type": "link", + "url": "https://github.com/MarquezProject/marquez/pull/2661", + "text": "#2661", + "unsafe": true + }, + { + "type": "text", + "text": " with minor comments, I think the " + }, + { + "type": "link", + "url": "https://github.com/MarquezProject/marquez/pull/2661/files#r1393820409", + "text": "enum defined in the db layer" + }, + { + "type": "text", + "text": " is one comment we’ll need to address before merging; otherwise solid work dude " + }, + { + "type": "emoji", + "name": "ok_hand", + "unicode": "1f44c" + } + ] + } + ] + } + ], + "team": "T01CWUYP5AR", + "edited": { + "user": "U01DCMDFHBK", + "ts": "1700037165.000000" + }, + "attachments": [ + { + "id": 1, + "footer_icon": "https://slack.github.com/static/img/favicon-neutral.png", + "ts": 1698327731, + "color": "36a64f", + "bot_id": "B01VA0FB340", + "app_unfurl_url": "https://github.com/MarquezProject/marquez/pull/2661", + "is_app_unfurl": true, + "app_id": "A01BP7R4KNY", + "fallback": "#2661 Runless events - consume job event", + "title": "#2661 Runless events - consume job event", + "title_link": "https://github.com/MarquezProject/marquez/pull/2661", + "footer": "", + "mrkdwn_in": [ + "text" + ] + } + ], + "reactions": [ + { + "name": "raised_hands", + "users": [ + "U02MK6YNAQ5", + "U01HNKK4XAM" + ], + "count": 2 + } + ] + }, + { + "client_msg_id": "e27ec408-535d-4a9b-a41b-c848a8cb286a", + "type": "message", + "text": "\nfun PR incoming", + "user": "U02S6F54MAB", + "ts": "1700004648.584649", + "blocks": [ + { + "type": "rich_text", + "block_id": "xxBfv", + "elements": [ + { + "type": "rich_text_section", + "elements": [ + { + "type": "link", + "url": "https://github.com/OpenLineage/OpenLineage/pull/2260" + }, + { + "type": "text", + "text": "\nfun PR incoming" + } + ] + } + ] + } + ], + "team": "T01CWUYP5AR", + "thread_ts": "1700004648.584649", + "reply_count": 5, + "reply_users_count": 4, + "latest_reply": "1700066753.852469", + "reply_users": [ + "U02S6F54MAB", + "U01RA9B5GG2", + "U02MK6YNAQ5", + "U01HNKK4XAM" + ], + "is_locked": false, + "subscribed": false, + "replies": [ + { + "type": "message", + "subtype": "thread_broadcast", + "text": "hey look, more fun\n", + "user": "U02S6F54MAB", + "ts": "1700040937.040239", + "thread_ts": "1700004648.584649", + "root": { + "client_msg_id": "e27ec408-535d-4a9b-a41b-c848a8cb286a", + "type": "message", + "text": "\nfun PR incoming", + "user": "U02S6F54MAB", + "ts": "1700004648.584649", + "blocks": [ + { + "type": "rich_text", + "block_id": "xxBfv", + "elements": [ + { + "type": "rich_text_section", + "elements": [ + { + "type": "link", + "url": "https://github.com/OpenLineage/OpenLineage/pull/2260" + }, + { + "type": "text", + "text": "\nfun PR incoming" + } + ] + } + ] + } + ], + "team": "T01CWUYP5AR", + "thread_ts": "1700004648.584649", + "reply_count": 5, + "reply_users_count": 4, + "latest_reply": "1700066753.852469", + "reply_users": [ + "U02S6F54MAB", + "U01RA9B5GG2", + "U02MK6YNAQ5", + "U01HNKK4XAM" + ], + "is_locked": false, + "subscribed": false + }, + "blocks": [ + { + "type": "rich_text", + "block_id": "yGPSQ", + "elements": [ + { + "type": "rich_text_section", + "elements": [ + { + "type": "text", + "text": "hey look, more fun\n" + }, + { + "type": "link", + "url": "https://github.com/OpenLineage/OpenLineage/pull/2263" + } + ] + } + ] + } + ], + "client_msg_id": "9fbbabe3-ab1f-4f16-8e55-28f574dd7eb5" + }, + { + "client_msg_id": "789d839b-2827-4542-b060-d72e2c135e3d", + "type": "message", + "text": "nice to have fun with you Jakub", + "user": "U01RA9B5GG2", + "ts": "1700042638.003809", + "blocks": [ + { + "type": "rich_text", + "block_id": "L8iP2", + "elements": [ + { + "type": "rich_text_section", + "elements": [ + { + "type": "text", + "text": "nice to have fun with you Jakub" + } + ] + } + ] + } + ], + "team": "T01CWUYP5AR", + "thread_ts": "1700004648.584649", + "parent_user_id": "U02S6F54MAB", + "reactions": [ + { + "name": "slightly_smiling_face", + "users": [ + "U02S6F54MAB", + "U01HNKK4XAM", + "U01DCMDFHBK" + ], + "count": 3 + } + ] + }, + { + "client_msg_id": "c388a688-15b4-4720-98c2-da976acec0a5", + "type": "message", + "text": "Can't wait to see it on the 1st January.", + "user": "U02MK6YNAQ5", + "ts": "1700044954.083129", + "blocks": [ + { + "type": "rich_text", + "block_id": "zN7Xj", + "elements": [ + { + "type": "rich_text_section", + "elements": [ + { + "type": "text", + "text": "Can't wait to see it on the 1st January." + } + ] + } + ] + } + ], + "team": "T01CWUYP5AR", + "thread_ts": "1700004648.584649", + "parent_user_id": "U02S6F54MAB" + }, + { + "client_msg_id": "1340B438-D041-4EAF-ADB3-604A3F1D302F", + "type": "message", + "text": "Ain’t no party like a dev ex improvement party", + "user": "U01HNKK4XAM", + "ts": "1700049363.338799", + "blocks": [ + { + "type": "rich_text", + "block_id": "6Yj06", + "elements": [ + { + "type": "rich_text_section", + "elements": [ + { + "type": "text", + "text": "Ain’t" + }, + { + "type": "text", + "text": " no party like a dev ex improvement party" + } + ] + } + ] + } + ], + "team": "T01CWUYP5AR", + "thread_ts": "1700004648.584649", + "parent_user_id": "U02S6F54MAB" + }, + { + "client_msg_id": "35ba5050-45f2-49f6-9126-aaa37a682593", + "type": "message", + "text": "Gentoo installation party is in similar category of fun", + "user": "U01RA9B5GG2", + "ts": "1700066753.852469", + "blocks": [ + { + "type": "rich_text", + "block_id": "vIT3H", + "elements": [ + { + "type": "rich_text_section", + "elements": [ + { + "type": "text", + "text": "Gentoo installation party is in similar category of fun" + } + ] + } + ] + } + ], + "team": "T01CWUYP5AR", + "thread_ts": "1700004648.584649", + "parent_user_id": "U02S6F54MAB" + } + ] + }, + { + "client_msg_id": "60159e06-de81-4b43-9edf-f404c32ddd38", + "type": "message", + "text": ":wave:", + "user": "U01RA9B5GG2", + "ts": "1699987988.642529", + "blocks": [ + { + "type": "rich_text", + "block_id": "YMxTi", + "elements": [ + { + "type": "rich_text_section", + "elements": [ + { + "type": "emoji", + "name": "wave", + "unicode": "1f44b" + } + ] + } + ] + } + ], + "team": "T01CWUYP5AR" + }, + { + "client_msg_id": "c329e984-0446-4e18-8fcf-74d020e30699", + "type": "message", + "text": ":ocean:", + "user": "U053LLVTHRN", + "ts": "1699982987.125549", + "blocks": [ + { + "type": "rich_text", + "block_id": "2BlqX", + "elements": [ + { + "type": "rich_text_section", + "elements": [ + { + "type": "emoji", + "name": "ocean", + "unicode": "1f30a" + } + ] + } + ] + } + ], + "team": "T01CWUYP5AR" + }, + { + "client_msg_id": "2aa5fdfa-5543-407e-a314-7623ef0c0763", + "type": "message", + "text": ":wave:", + "user": "U01DCMDFHBK", + "ts": "1699982333.485469", + "blocks": [ + { + "type": "rich_text", + "block_id": "YMxTi", + "elements": [ + { + "type": "rich_text_section", + "elements": [ + { + "type": "emoji", + "name": "wave", + "unicode": "1f44b" + } + ] + } + ] + } + ], + "team": "T01CWUYP5AR" + }, + { + "client_msg_id": "818CA217-10D2-492F-B387-499128576860", + "type": "message", + "text": ":wave: ", + "user": "U01DCLP0GU9", + "ts": "1699982322.990799", + "blocks": [ + { + "type": "rich_text", + "block_id": "msKVM", + "elements": [ + { + "type": "rich_text_section", + "elements": [ + { + "type": "emoji", + "name": "wave", + "unicode": "1f44b" + }, + { + "type": "text", + "text": " " + } + ] + } + ] + } + ], + "team": "T01CWUYP5AR" + }, + { + "client_msg_id": "4225176b-1766-4f0f-9d34-e2c808164b02", + "type": "message", + "text": ":wave:", + "user": "U02LXF3HUN7", + "ts": "1699982179.651129", + "blocks": [ + { + "type": "rich_text", + "block_id": "YMxTi", + "elements": [ + { + "type": "rich_text_section", + "elements": [ + { + "type": "emoji", + "name": "wave", + "unicode": "1f44b" + } + ] + } + ] + } + ], + "team": "T01CWUYP5AR" + }, + { + "type": "message", + "subtype": "channel_join", + "ts": "1699982042.414079", + "user": "U053LLVTHRN", + "text": "<@U053LLVTHRN> has joined the channel", + "inviter": "U01HNKK4XAM" + }, + { + "client_msg_id": "0bea58f3-10b1-4972-ad6a-0a002ae61854", + "type": "message", + "text": ":wave:", + "user": "U02S6F54MAB", + "ts": "1699982037.267109", + "blocks": [ + { + "type": "rich_text", + "block_id": "YMxTi", + "elements": [ + { + "type": "rich_text_section", + "elements": [ + { + "type": "emoji", + "name": "wave", + "unicode": "1f44b" + } + ] + } + ] + } + ], + "team": "T01CWUYP5AR" + }, + { + "type": "message", + "subtype": "channel_join", + "ts": "1699982026.646699", + "user": "U05KKM07PJP", + "text": "<@U05KKM07PJP> has joined the channel", + "inviter": "U01HNKK4XAM" + }, + { + "type": "message", + "subtype": "channel_join", + "ts": "1699982026.554329", + "user": "U01DCMDFHBK", + "text": "<@U01DCMDFHBK> has joined the channel", + "inviter": "U01HNKK4XAM" + }, + { + "type": "message", + "subtype": "channel_join", + "ts": "1699982026.462039", + "user": "U02LXF3HUN7", + "text": "<@U02LXF3HUN7> has joined the channel", + "inviter": "U01HNKK4XAM" + }, + { + "type": "message", + "subtype": "channel_join", + "ts": "1699982026.350149", + "user": "U02S6F54MAB", + "text": "<@U02S6F54MAB> has joined the channel", + "inviter": "U01HNKK4XAM" + }, + { + "type": "message", + "subtype": "channel_join", + "ts": "1699982026.266479", + "user": "U02MK6YNAQ5", + "text": "<@U02MK6YNAQ5> has joined the channel", + "inviter": "U01HNKK4XAM" + }, + { + "type": "message", + "subtype": "channel_join", + "ts": "1699982026.191829", + "user": "U01DCLP0GU9", + "text": "<@U01DCLP0GU9> has joined the channel", + "inviter": "U01HNKK4XAM" + }, + { + "type": "message", + "subtype": "channel_join", + "ts": "1699981990.850589", + "user": "U01RA9B5GG2", + "text": "<@U01RA9B5GG2> has joined the channel", + "inviter": "U01HNKK4XAM" + }, + { + "type": "message", + "subtype": "channel_join", + "ts": "1699981986.459199", + "user": "U01HNKK4XAM", + "text": "<@U01HNKK4XAM> has joined the channel" + } +] \ No newline at end of file diff --git a/slack-archive/data/channels.json b/slack-archive/data/channels.json new file mode 100644 index 0000000..4248056 --- /dev/null +++ b/slack-archive/data/channels.json @@ -0,0 +1,570 @@ +[ + { + "id": "C01CK9T7HKR", + "name": "general", + "is_channel": true, + "is_group": false, + "is_im": false, + "is_mpim": false, + "is_private": false, + "created": 1603242062, + "is_archived": false, + "is_general": true, + "unlinked": 0, + "name_normalized": "general", + "is_shared": false, + "is_org_shared": false, + "is_pending_ext_shared": false, + "pending_shared": [], + "context_team_id": "T01CWUYP5AR", + "updated": 1690953251304, + "parent_conversation": null, + "creator": "U01DCLP0GU9", + "is_ext_shared": false, + "shared_team_ids": [ + "T01CWUYP5AR" + ], + "pending_connected_team_ids": [], + "is_member": true, + "topic": { + "value": "OpenLineage spec discussion.\nSee also: \n2023 Ecosystem Survey: ", + "creator": "U02LXF3HUN7", + "last_set": 1684872900 + }, + "purpose": { + "value": "This is the one channel that will always include everyone. It’s a great spot for announcements and team-wide conversations.", + "creator": "U01DCLP0GU9", + "last_set": 1603242062 + }, + "properties": { + "canvas": { + "file_id": "F05KFAKSTHV", + "is_empty": true, + "quip_thread_id": "BZB9AAVq0C6" + } + }, + "previous_names": [], + "num_members": 1189 + }, + { + "id": "C01NAFMBVEY", + "name": "mark-grover", + "is_channel": true, + "is_group": false, + "is_im": false, + "is_mpim": false, + "is_private": false, + "created": 1613391683, + "is_archived": true, + "is_general": false, + "unlinked": 1613391683, + "name_normalized": "mark-grover", + "is_shared": false, + "is_org_shared": false, + "is_pending_ext_shared": false, + "pending_shared": [], + "context_team_id": "T01CWUYP5AR", + "updated": 1613391683292, + "frozen_reason": "connection_severed", + "parent_conversation": null, + "creator": "U01HNSHB2H4", + "is_ext_shared": false, + "shared_team_ids": [ + "T01CWUYP5AR" + ], + "pending_connected_team_ids": [], + "is_member": false, + "topic": { + "value": "", + "creator": "", + "last_set": 0 + }, + "purpose": { + "value": "", + "creator": "", + "last_set": 0 + }, + "previous_names": [], + "num_members": 0 + }, + { + "id": "C030F1J0264", + "name": "dagster-integration", + "is_channel": true, + "is_group": false, + "is_im": false, + "is_mpim": false, + "is_private": false, + "created": 1642818372, + "is_archived": false, + "is_general": false, + "unlinked": 0, + "name_normalized": "dagster-integration", + "is_shared": false, + "is_org_shared": false, + "is_pending_ext_shared": false, + "pending_shared": [], + "context_team_id": "T01CWUYP5AR", + "updated": 1642818373220, + "parent_conversation": null, + "creator": "U025D1JDTRB", + "is_ext_shared": false, + "shared_team_ids": [ + "T01CWUYP5AR" + ], + "pending_connected_team_ids": [], + "is_member": true, + "topic": { + "value": "", + "creator": "", + "last_set": 0 + }, + "purpose": { + "value": "OpenLineage-Dagster integration discussion", + "creator": "U025D1JDTRB", + "last_set": 1642818373 + }, + "previous_names": [], + "num_members": 47 + }, + { + "id": "C04E3Q18RR9", + "name": "open-lineage-plus-bacalhau", + "is_channel": true, + "is_group": false, + "is_im": false, + "is_mpim": false, + "is_private": false, + "created": 1670404449, + "is_archived": false, + "is_general": false, + "unlinked": 0, + "name_normalized": "open-lineage-plus-bacalhau", + "is_shared": false, + "is_org_shared": false, + "is_pending_ext_shared": false, + "pending_shared": [], + "context_team_id": "T01CWUYP5AR", + "updated": 1670404449626, + "parent_conversation": null, + "creator": "U044VPCNMDX", + "is_ext_shared": false, + "shared_team_ids": [ + "T01CWUYP5AR" + ], + "pending_connected_team_ids": [], + "is_member": false, + "topic": { + "value": "", + "creator": "", + "last_set": 0 + }, + "purpose": { + "value": "", + "creator": "", + "last_set": 0 + }, + "previous_names": [], + "num_members": 10 + }, + { + "id": "C04JPTTC876", + "name": "spec-compliance", + "is_channel": true, + "is_group": false, + "is_im": false, + "is_mpim": false, + "is_private": false, + "created": 1673531750, + "is_archived": false, + "is_general": false, + "unlinked": 0, + "name_normalized": "spec-compliance", + "is_shared": false, + "is_org_shared": false, + "is_pending_ext_shared": false, + "pending_shared": [], + "context_team_id": "T01CWUYP5AR", + "updated": 1673550344241, + "parent_conversation": null, + "creator": "U01DCLP0GU9", + "is_ext_shared": false, + "shared_team_ids": [ + "T01CWUYP5AR" + ], + "pending_connected_team_ids": [], + "is_member": true, + "topic": { + "value": "Key points/decisions -\n", + "creator": "U0323HG8C8H", + "last_set": 1673550344 + }, + "purpose": { + "value": "", + "creator": "", + "last_set": 0 + }, + "previous_names": [], + "num_members": 25 + }, + { + "id": "C04QSV0GG23", + "name": "providence-meetup", + "is_channel": true, + "is_group": false, + "is_im": false, + "is_mpim": false, + "is_private": false, + "created": 1677267471, + "is_archived": true, + "is_general": false, + "unlinked": 0, + "name_normalized": "providence-meetup", + "is_shared": false, + "is_org_shared": false, + "is_pending_ext_shared": false, + "pending_shared": [], + "context_team_id": "T01CWUYP5AR", + "updated": 1680640801888, + "parent_conversation": null, + "creator": "U02LXF3HUN7", + "is_ext_shared": false, + "shared_team_ids": [ + "T01CWUYP5AR" + ], + "pending_connected_team_ids": [], + "is_member": false, + "topic": { + "value": "", + "creator": "", + "last_set": 0 + }, + "purpose": { + "value": "More info: \nSign up: ", + "creator": "U02LXF3HUN7", + "last_set": 1677269229 + }, + "previous_names": [], + "num_members": 0 + }, + { + "id": "C04THH1V90X", + "name": "data-council-meetup", + "is_channel": true, + "is_group": false, + "is_im": false, + "is_mpim": false, + "is_private": false, + "created": 1678822970, + "is_archived": false, + "is_general": false, + "unlinked": 0, + "name_normalized": "data-council-meetup", + "is_shared": false, + "is_org_shared": false, + "is_pending_ext_shared": false, + "pending_shared": [], + "context_team_id": "T01CWUYP5AR", + "updated": 1678822970441, + "parent_conversation": null, + "creator": "U02LXF3HUN7", + "is_ext_shared": false, + "shared_team_ids": [ + "T01CWUYP5AR" + ], + "pending_connected_team_ids": [], + "is_member": false, + "topic": { + "value": "", + "creator": "", + "last_set": 0 + }, + "purpose": { + "value": "A channel for syncing up about travel plans, etc.", + "creator": "U02LXF3HUN7", + "last_set": 1678822970 + }, + "previous_names": [], + "num_members": 4 + }, + { + "id": "C051C93UZK9", + "name": "nyc-meetup", + "is_channel": true, + "is_group": false, + "is_im": false, + "is_mpim": false, + "is_private": false, + "created": 1680631804, + "is_archived": false, + "is_general": false, + "unlinked": 0, + "name_normalized": "nyc-meetup", + "is_shared": false, + "is_org_shared": false, + "is_pending_ext_shared": false, + "pending_shared": [], + "context_team_id": "T01CWUYP5AR", + "updated": 1680631804998, + "parent_conversation": null, + "creator": "U02LXF3HUN7", + "is_ext_shared": false, + "shared_team_ids": [ + "T01CWUYP5AR" + ], + "pending_connected_team_ids": [], + "is_member": true, + "topic": { + "value": "", + "creator": "", + "last_set": 0 + }, + "purpose": { + "value": "For coordinating on travel, etc.", + "creator": "U02LXF3HUN7", + "last_set": 1680631804 + }, + "previous_names": [], + "num_members": 7 + }, + { + "id": "C055GGUFMHQ", + "name": "boston-meetup", + "is_channel": true, + "is_group": false, + "is_im": false, + "is_mpim": false, + "is_private": false, + "created": 1682700563, + "is_archived": false, + "is_general": false, + "unlinked": 0, + "name_normalized": "boston-meetup", + "is_shared": false, + "is_org_shared": false, + "is_pending_ext_shared": false, + "pending_shared": [], + "context_team_id": "T01CWUYP5AR", + "updated": 1682700563376, + "parent_conversation": null, + "creator": "U02LXF3HUN7", + "is_ext_shared": false, + "shared_team_ids": [ + "T01CWUYP5AR" + ], + "pending_connected_team_ids": [], + "is_member": true, + "topic": { + "value": "", + "creator": "", + "last_set": 0 + }, + "purpose": { + "value": "", + "creator": "", + "last_set": 0 + }, + "previous_names": [], + "num_members": 7 + }, + { + "id": "C056YHEU680", + "name": "sf-meetup", + "is_channel": true, + "is_group": false, + "is_im": false, + "is_mpim": false, + "is_private": false, + "created": 1683210950, + "is_archived": false, + "is_general": false, + "unlinked": 0, + "name_normalized": "sf-meetup", + "is_shared": false, + "is_org_shared": false, + "is_pending_ext_shared": false, + "pending_shared": [], + "context_team_id": "T01CWUYP5AR", + "updated": 1683211020224, + "parent_conversation": null, + "creator": "U02LXF3HUN7", + "is_ext_shared": false, + "shared_team_ids": [ + "T01CWUYP5AR" + ], + "pending_connected_team_ids": [], + "is_member": true, + "topic": { + "value": "", + "creator": "", + "last_set": 0 + }, + "purpose": { + "value": "Please join the meetup group: ", + "creator": "U02LXF3HUN7", + "last_set": 1683211020 + }, + "previous_names": [], + "num_members": 11 + }, + { + "id": "C05N442RQUA", + "name": "toronto-meetup", + "is_channel": true, + "is_group": false, + "is_im": false, + "is_mpim": false, + "is_private": false, + "created": 1692210102, + "is_archived": false, + "is_general": false, + "unlinked": 0, + "name_normalized": "toronto-meetup", + "is_shared": false, + "is_org_shared": false, + "is_pending_ext_shared": false, + "pending_shared": [], + "context_team_id": "T01CWUYP5AR", + "updated": 1692210102912, + "parent_conversation": null, + "creator": "U02LXF3HUN7", + "is_ext_shared": false, + "shared_team_ids": [ + "T01CWUYP5AR" + ], + "pending_connected_team_ids": [], + "is_member": true, + "topic": { + "value": "", + "creator": "", + "last_set": 0 + }, + "purpose": { + "value": "", + "creator": "", + "last_set": 0 + }, + "previous_names": [], + "num_members": 9 + }, + { + "id": "C05PD7VJ52S", + "name": "london-meetup", + "is_channel": true, + "is_group": false, + "is_im": false, + "is_mpim": false, + "is_private": false, + "created": 1692978724, + "is_archived": false, + "is_general": false, + "unlinked": 0, + "name_normalized": "london-meetup", + "is_shared": false, + "is_org_shared": false, + "is_pending_ext_shared": false, + "pending_shared": [], + "context_team_id": "T01CWUYP5AR", + "updated": 1692978724912, + "parent_conversation": null, + "creator": "U02LXF3HUN7", + "is_ext_shared": false, + "shared_team_ids": [ + "T01CWUYP5AR" + ], + "pending_connected_team_ids": [], + "is_member": true, + "topic": { + "value": "", + "creator": "", + "last_set": 0 + }, + "purpose": { + "value": "", + "creator": "", + "last_set": 0 + }, + "previous_names": [], + "num_members": 6 + }, + { + "id": "C05U3UC85LM", + "name": "gx-integration", + "is_channel": true, + "is_group": false, + "is_im": false, + "is_mpim": false, + "is_private": false, + "created": 1695836283, + "is_archived": false, + "is_general": false, + "unlinked": 0, + "name_normalized": "gx-integration", + "is_shared": false, + "is_org_shared": false, + "is_pending_ext_shared": false, + "pending_shared": [], + "context_team_id": "T01CWUYP5AR", + "updated": 1695836283474, + "parent_conversation": null, + "creator": "U02LXF3HUN7", + "is_ext_shared": false, + "shared_team_ids": [ + "T01CWUYP5AR" + ], + "pending_connected_team_ids": [], + "is_member": true, + "topic": { + "value": "", + "creator": "", + "last_set": 0 + }, + "purpose": { + "value": "", + "creator": "", + "last_set": 0 + }, + "previous_names": [], + "num_members": 7 + }, + { + "id": "C065PQ4TL8K", + "name": "dev-discuss", + "is_channel": true, + "is_group": false, + "is_im": false, + "is_mpim": false, + "is_private": false, + "created": 1699981986, + "is_archived": false, + "is_general": false, + "unlinked": 0, + "name_normalized": "dev-discuss", + "is_shared": false, + "is_org_shared": false, + "is_pending_ext_shared": false, + "pending_shared": [], + "context_team_id": "T01CWUYP5AR", + "updated": 1699981986353, + "parent_conversation": null, + "creator": "U01HNKK4XAM", + "is_ext_shared": false, + "shared_team_ids": [ + "T01CWUYP5AR" + ], + "pending_connected_team_ids": [], + "is_member": true, + "topic": { + "value": "", + "creator": "", + "last_set": 0 + }, + "purpose": { + "value": "", + "creator": "", + "last_set": 0 + }, + "previous_names": [], + "num_members": 9 + } +] \ No newline at end of file diff --git a/slack-archive/data/emojis.json b/slack-archive/data/emojis.json new file mode 100644 index 0000000..8934469 --- /dev/null +++ b/slack-archive/data/emojis.json @@ -0,0 +1,30 @@ +{ + "bowtie": "https://emoji.slack-edge.com/T01CWUYP5AR/bowtie/f3ec6f2bb0.png", + "squirrel": "https://emoji.slack-edge.com/T01CWUYP5AR/squirrel/465f40c0e0.png", + "glitch_crab": "https://emoji.slack-edge.com/T01CWUYP5AR/glitch_crab/db049f1f9c.png", + "piggy": "https://emoji.slack-edge.com/T01CWUYP5AR/piggy/b7762ee8cd.png", + "cubimal_chick": "https://emoji.slack-edge.com/T01CWUYP5AR/cubimal_chick/85961c43d7.png", + "dusty_stick": "https://emoji.slack-edge.com/T01CWUYP5AR/dusty_stick/6177a62312.png", + "slack": "https://emoji.slack-edge.com/T01CWUYP5AR/slack/7d462d2443.png", + "pride": "https://emoji.slack-edge.com/T01CWUYP5AR/pride/56b1bd3388.png", + "thumbsup_all": "https://emoji.slack-edge.com/T01CWUYP5AR/thumbsup_all/50096a1020.png", + "slack_call": "https://emoji.slack-edge.com/T01CWUYP5AR/slack_call/b81fffd6dd.png", + "shipit": "alias:squirrel", + "white_square": "alias:white_large_square", + "black_square": "alias:black_large_square", + "simple_smile": "https://a.slack-edge.com/80588/img/emoji_2017_12_06/apple/simple_smile.png", + "raito": "https://emoji.slack-edge.com/T01CWUYP5AR/raito/99218314dd778726.png", + "gratitude-arigatou-gozaimasu": "https://emoji.slack-edge.com/T01CWUYP5AR/gratitude-arigatou-gozaimasu/d6322e811f6c8071.png", + "gratitude-asl": "https://emoji.slack-edge.com/T01CWUYP5AR/gratitude-asl/d6322e811f6c8071.gif", + "gratitude-danke": "https://emoji.slack-edge.com/T01CWUYP5AR/gratitude-danke/d6322e811f6c8071.png", + "gratitude-gamsahamnida": "https://emoji.slack-edge.com/T01CWUYP5AR/gratitude-gamsahamnida/d6322e811f6c8071.png", + "gratitude-gracias": "https://emoji.slack-edge.com/T01CWUYP5AR/gratitude-gracias/d6322e811f6c8071.png", + "gratitude-grazie": "https://emoji.slack-edge.com/T01CWUYP5AR/gratitude-grazie/d6322e811f6c8071.png", + "gratitude-merci": "https://emoji.slack-edge.com/T01CWUYP5AR/gratitude-merci/d6322e811f6c8071.png", + "gratitude-obrigada": "https://emoji.slack-edge.com/T01CWUYP5AR/gratitude-obrigada/d6322e811f6c8071.png", + "gratitude-obrigado": "https://emoji.slack-edge.com/T01CWUYP5AR/gratitude-obrigado/d6322e811f6c8071.png", + "gratitude-spasibo": "https://emoji.slack-edge.com/T01CWUYP5AR/gratitude-spasibo/d6322e811f6c8071.png", + "gratitude-thank-you": "https://emoji.slack-edge.com/T01CWUYP5AR/gratitude-thank-you/d6322e811f6c8071.png", + "gratitude-xiexie": "https://emoji.slack-edge.com/T01CWUYP5AR/gratitude-xiexie/d6322e811f6c8071.png", + "gratitude-xiexieni": "https://emoji.slack-edge.com/T01CWUYP5AR/gratitude-xiexieni/d6322e811f6c8071.png" +} \ No newline at end of file diff --git a/slack-archive/data/search.js b/slack-archive/data/search.js new file mode 100644 index 0000000..8fe7696 --- /dev/null +++ b/slack-archive/data/search.js @@ -0,0 +1 @@ +window.search_data = {"channels":{"C01CK9T7HKR":"general","C01NAFMBVEY":"mark-grover","C030F1J0264":"dagster-integration","C04E3Q18RR9":"open-lineage-plus-bacalhau","C04JPTTC876":"spec-compliance","C04QSV0GG23":"providence-meetup","C04THH1V90X":"data-council-meetup","C051C93UZK9":"nyc-meetup","C055GGUFMHQ":"boston-meetup","C056YHEU680":"sf-meetup","C05N442RQUA":"toronto-meetup","C05PD7VJ52S":"london-meetup","C05U3UC85LM":"gx-integration","C065PQ4TL8K":"dev-discuss"},"users":{"U066S97A90C":"rwojcik","U066CNW85D3":"karthik.nandagiri","U066HKFCHUG":"naresh.naresh36","U05T8BJD4DU":"jasonyip","U02LXF3HUN7":"michael282","U05TU0U224A":"rodrigo.maia","U05NMJ0NBUK":"lance.dacey2","U05J9LZ355L":"yannick.libert.partne","U05JBHLPY8K":"athityakumar","U0635GK8Y14":"david.goss","U062Q95A1FG":"n.priya88","U063YP6UJJ0":"fangmik","U04AZ7992SU":"john490","U062WLFMRTP":"hloomba","U05CAULTYG2":"kkandaswamy","U06315TMT61":"splicer9904","U0625RZ7KR9":"kpraveen420","U05KCF3EEUR":"savansharan_navalgi","U03D8K119LJ":"matthewparas2020","U0616K9TSTZ":"ankit.goods10","U04EZ2LPDV4":"anirudh.shrinivason","U05HK41VCH1":"madhav.kakumani","U05QL7LN2GH":"jeevan","U021QJMRP47":"drew215","U01HVNU6A4C":"mars","U01DCLP0GU9":"julien","U05FLJE4GDU":"damien.hawes","U05TZE47F2S":"slack1950","U05A1D80QKF":"suraj.gupta","U05SMTVPPL3":"sangeeta","U05HFGKEYVB":"juan_luis_cano","U05SQGH8DV4":"sarathch","U05K8F1T887":"terese","U055N2GRT4P":"tatiana.alchueyr","U05QNRSQW1E":"sarwatfatimam","U0595Q78HUG":"gaborjbernat","U05HBLE7YPL":"abdallah","U0323HG8C8H":"sheeri.cabral","U05NGJ8AM8X":"yunhe52203334","U05EC8WB74N":"mbarrien","U05JY6MN8MS":"githubopenlineageissu","U05PVS8GRJ6":"josdotso","U05J5GRKY10":"george.polychronopoul","U05TQPZ4R4L":"aaruna6","U05Q3HT6PBR":"kevin","U05SXDWVA7K":"kgkwiz","U01HNKK4XAM":"harel.shein","U05QHG1NJ8J":"mike474","U01RA9B5GG2":"maciej.obuchowski","U0620HU51HA":"sicotte.jason","U05U9K21LSG":"bill","U02S6F54MAB":"jakub.dardzinski","U05U9929K3N":"don","U01DCMDFHBK":"willy","U02MK6YNAQ5":"pawel.leszczynski","U053LLVTHRN":"ross769","U05KKM07PJP":"peter.hicks"},"messages":{"C01CK9T7HKR":[{"m":"Hi Everyone, first of all - big shout to all contributors - You do amazing job here.\nI want to use OpenLineage in our project - to do so I want to setup some POC and experiment with possibilities library provides - I start working on sample from the conference talk: but when I go into spark transformation after staring context with openlineage I have issues with _SessionHiveMetaStoreClient on section 3_- does anyone has other plain sample to play with, to not setup everything from scratch?","u":"U066S97A90C","t":"1700568128.192669"},{"m":"Hi So we can use openlineage to identify column level lineage with Airflow , Spark? will it also allow to connect to Power BI and derive the downstream column lineage ?","u":"U066CNW85D3","t":"1700456258.614309"},{"m":"what are the pros and cons of OL. we often talk about positives to market it but what are the pain points using OL,how it's addressing user issues?","u":"U066HKFCHUG","t":"1700064658.956769"},{"m":"Can anyone tell me why OL is better than other competitors if you can provide an analysis that would be great","u":"U066HKFCHUG","t":"1700064564.825909"},{"m":"Hi\nCan anyone point me to the deck on how Airflow can be integrated using Openlineage?","u":"U066HKFCHUG","t":"1700050644.509419"},{"m":"<@U02MK6YNAQ5> I diff CreateReplaceDatasetBuilder.java and CreateReplaceOutputDatasetBuilder.java and they are the same except for the class name, so I am not sure what is causing the change. I also realize you don't have a test case for ADLS","u":"U05T8BJD4DU","t":"1699863517.394909"},{"m":"<@U02MK6YNAQ5> I went back to 1.4.1, output does show adls location. But environment facet is gone in 1.4.1. It shows up in 1.5.0 but namespace is back to dbfs....","u":"U05T8BJD4DU","t":"1699862442.876219"},{"m":"Databricks needs to be re-written in a way that supports Databricks it seems like","u":"U05T8BJD4DU","t":"1699691444.226989"},{"m":"<@U02MK6YNAQ5> this is why if create a table with adls location it won't show input and output:\n\n\n\nBecause the catalog object is not there.","u":"U05T8BJD4DU","t":"1699691373.531469"},{"m":"<@U02MK6YNAQ5> regarding to , OL is parsing out the table location in Hive metastore, it is the location of the table in the catalog and not the physical location of the data. It is both right and wrong because it is a table, just it is an external table.\n\n","u":"U05T8BJD4DU","t":"1699647945.224489"},{"m":"\nFriendly reminder: this month’s TSC meeting, open to all, is tomorrow at 10 am PT: ","u":"U02LXF3HUN7","t":"1699465494.687309"},{"m":"Has anyone here tried OpenLineage with Spark on Amazon EMR?","u":"U05TU0U224A","t":"1699465132.534889"},{"m":"if I have a dataset on adls gen2 which synapse connects to as an external delta table, is that the use case of a symlink dataset? the delta table is connected to by PBI and by Synapse, but the underlying data is exactly the same","u":"U05NMJ0NBUK","t":"1699372165.804069"},{"m":"Hi all, we (I work with <@U05VDHJJ9T7> and <@U05HBLE7YPL>) have a quick question regarding the spark integration:\nif a spark app contains several jobs, they will be named \"my_spark_app_name.job1\" and \"my_spark_app_name.job2\"\neg:\nspark_job.collect_limit\nspark_job.map_partitions_parallel_collection\n\nIf I understood correctly, the spark integration maps one Spark job to a single OpenLineage Job, and the application itself should be assigned a Run id at startup and each job that executes will report the application's Run id as its parent job run (taken from: ).\n\nIn our case, the app Run Id is never created, and the jobs runs don't contain any parent facets. We tested it with a recent integration version in 1.4.1 and also an older one (0.26.0).\nDid we miss something in the OL spark integration config?","u":"U05J9LZ355L","t":"1699355029.839029"},{"m":"Hey team! :wave:\n\nWe're trying to use openlineage-flink, and would like provide the `openlineage.transport.type=http` and configure other transport configs, but we're not able to find sufficient docs (tried ) on where/how these configs can be provided.\n\nFor example, in spark, the changes mostly were delegated to the spark-submit command like\n```spark-submit --conf \"spark.extraListeners=io.openlineage.spark.agent.OpenLineageSparkListener\" \\\n --packages \"io.openlineage:openlineage-spark:<spark-openlineage-version>\" \\\n --conf \"spark.openlineage.transport.url=http://{openlineage.client.host}/api/v1/namespaces/spark_integration/\" \\\n --class com.mycompany.MySparkApp my_application.jar```\nAnd the `OpenLineageSparkListener` has a method to retrieve the provided spark confs as an object in the ArgumentParser. Similarly, looking for some pointers on how the openlineage.transport configs can be provided to `OpenLineageFlinkJobListener` & how the flink listener parses/uses these configs\n\nTIA! :smile:","u":"U05JBHLPY8K","t":"1699266123.453379"},{"m":":wave: I raised a PR off the back of some Marquez conversations a while back to try and clarify how names of Snowflake objects should be expressed in OL events. I used as a guide, but also I appreciate there are other OL producers that involve Snowflake too (Airflow? dbt?). Any feedback on this would be appreciated!","u":"U0635GK8Y14","t":"1699261422.618719"},{"m":"Hi Team , we are trying to customize the events by writing custom lineage listener extending OpenLineageSparkListener, but would need some direction how to capture the events","u":"U062Q95A1FG","t":"1699096090.087359"},{"m":"\nThis month’s TSC meeting (open to all) is next Thursday the 9th at 10am PT. On the agenda:\n• announcements\n• recent releases\n• recent additions to the Flink integration by <@U05QA2D1XNV> \n• recent additions to the Spark integration by <@U02MK6YNAQ5> \n• updates on proposals by <@U01DCLP0GU9> \n• discussion topics\n• open discussion\nMore info and the meeting link can be found on the . All are welcome! Do you have a discussion topic, use case or integration you’d like to demo? DM me to be added to the agenda.","u":"U02LXF3HUN7","t":"1699027207.361229"},{"m":"actually, it shows up in one of the RUNNING now... behavior is consistent between 11.3 and 13.3, thanks for fixing this issue","u":"U05T8BJD4DU","t":"1698999491.798599"},{"m":"<@U02MK6YNAQ5> I tested 1.5.0, it works great now, but the environment facets is gone in START... which I very much want it.. any thoughts?","u":"U05T8BJD4DU","t":"1698950958.157459"},{"m":"\nWe released OpenLineage 1.5.0, including:\n• by <@U05QA2D1XNV> \n• by <@U02MK6YNAQ5> \n• `rdd``toDF` by <@U02MK6YNAQ5> \n• by <@U02S6F54MAB> \n• by <@U02S6F54MAB> \n• bug fixes, tests, infra fixes, doc changes, and more.\nThanks to all the contributors, including new contributor <@U05VDHJJ9T7>!\n*Release:* \n*Changelog:* \n*Commit history:* \n*Maven:* \n*PyPI:* ","u":"U02LXF3HUN7","t":"1698940800.306129"},{"m":"I am looking to send OpenLineage events to an AWS API Gateway endpoint from an AWS MWAA instance. The problem is that all requests to AWS services need to be signed with SigV4, and using API Gateway with IAM authentication would require requests to API Gateway be signed with SigV4. Would the best way to do so be to just modify the python client HTTP transport to include a new config option for signing emitted OpenLineage events with SigV4? Are there any alternatives?","u":"U063YP6UJJ0","t":"1698885038.172079"},{"m":"Hi team :wave: , we’re finding that for our Spark jobs we are almost always getting some junk characters in our dataset names. We’ve pushed the regex filter to its limits and would like to extend the logic of deriving the dataset name in openlineage-spark (currently on `1.4.1`). I seem to recall hearing we could do this by implementing our own `LogicalPlanVisitor` or something along those lines? Is that still the recommended approach and if so would this be possible to implement in Scala vs. Java (scala noob here :simple_smile:)","u":"U04AZ7992SU","t":"1698882039.335099"},{"m":"\nThe October 2023 issue of is available now! to get in directly in your inbox each month.","u":"U02LXF3HUN7","t":"1698859749.531699"},{"m":"\nI’m opening a vote to release OpenLineage 1.5.0, including:\n• support for Cassandra Connectors lineage in the Flink integration\n• support for Databricks Runtime 13.3 in the Spark integration\n• support for `rdd` and `toDF` operations from the Spark Scala API in Spark\n• lowered requirements for attrs and requests packages in the Airflow integration\n• lazy rendering of yaml configs in the dbt integration\n• bug fixes, tests, infra fixes, doc changes, and more.\nThree +1s from committers will authorize an immediate release.","u":"U02LXF3HUN7","t":"1698852883.658009"},{"m":"one question if someone is around - when im keeping both `openlineage-airflow` and `apache-airflow-providers-openlineage` in my requirement file, i see the following error -\n``` from openlineage.airflow.extractors import Extractors\nModuleNotFoundError: No module named 'openlineage.airflow'```\nany thoughts?","u":"U062WLFMRTP","t":"1698778838.540239"},{"m":":wave: Hi team, cross-posting from the Marquez Channel in case anyone here has a better idea of the spec\n\n> For most of our lineage extractors in airflow, we are using the rust sql parser from openlineage-sql to extract table lineage via sql statements. When errors occur we are adding an `extractionError` run facet similar to what is being done . I’m finding in the case that multiple statements were extracted but one failed to parse while many others were successful, the lineage for these runs doesn’t appear as expected in Marquez. Is there any logic around the `extractionError` run facet that could be causing this? It seems reasonable to assume that we might take this to mean the entire run event is invalid if we have any extraction errors. \n> \n> I would still expect to see the other lineage we sent for the run but am instead just seeing the `extractionError` in the marquez UI, in the database, runs with an `extractionError` facet don’t seem to make it to the `job_versions_io_mapping` table","u":"U04AZ7992SU","t":"1698706303.956579"},{"m":"I realize in Spark 3.4+, some job ids don't have a start event. What part of the code is responsible for triggering the START and COMPLETE event","u":"U05T8BJD4DU","t":"1698563188.319939"},{"m":"Hello, has anyone run into similar error as posted in this github open issues[] while setting up marquez on an EC2 Instance, would appreciate any help to get past the errors","u":"U05CAULTYG2","t":"1698440472.145489"},{"m":"referencing to conversation - what it takes to move to openlineage provider package from openlineage-airflow. Im updating Airflow to 2.7.2 but moving off of openlineage-airflow to provider package Im trying to estimate the amount of work it takes, any thoughts? reading change_logs I dont think its too much of a change but please share your thoughts and if somewhere its drafted please do share that as well","u":"U062WLFMRTP","t":"1698429543.349989"},{"m":"Hi People, actually I want to intercept the OpenLineage spark events right after the job ends and before they are emitted, so that I can add some extra information to the events or remove some information that I don't want.\nIs there any way of doing this? Can someone please help me","u":"U06315TMT61","t":"1698408752.647169"},{"m":"*Spark Integration Logs*\nHey There\nAre these events skipped because it's not supported or it's configured somewhere?\n`23/10/27 08:25:58 INFO SparkSQLExecutionContext: OpenLineage received Spark event that is configured to be skipped: SparkListenerSQLExecutionStart`\n`23/10/27 08:25:58 INFO SparkSQLExecutionContext: OpenLineage received Spark event that is configured to be skipped: SparkListenerSQLExecutionEnd`","u":"U05TU0U224A","t":"1698400165.662489"},{"m":"Im upgrading the version from openlineage-airflow==0.24.0 to openlineage-airflow 1.4.1 but im seeing the following error, any help is appreciated","u":"U062WLFMRTP","t":"1698340358.557159"},{"m":"Hello Team","u":"U062WLFMRTP","t":"1698340277.847709"},{"m":"Hi I want to customise the events which comes from Openlineage spark . Can some one give some information","u":"U062Q95A1FG","t":"1698315220.142929"},{"m":"Hi,\n\nWe are using openlineage spark connector. We have used spark 3.2 and scala 2.12 so far. We have triggered a new job with Spark 3.4 and scala 2.13 and faced below exception.\n\n\n```java.lang.NoSuchMethodError: 'scala.collection.Seq org.apache.spark.sql.catalyst.plans.logical.LogicalPlan.map(scala.Function1)'\n\tat io.openlineage.spark.agent.lifecycle.OpenLineageRunEventBuilder.lambda$buildInputDatasets$6(OpenLineageRunEventBuilder.java:341)\n\tat java.base/java.util.Optional.map(Optional.java:265)\n\tat io.openlineage.spark.agent.lifecycle.OpenLineageRunEventBuilder.buildInputDatasets(OpenLineageRunEventBuilder.java:339)\n\tat io.openlineage.spark.agent.lifecycle.OpenLineageRunEventBuilder.populateRun(OpenLineageRunEventBuilder.java:295)\n\tat io.openlineage.spark.agent.lifecycle.OpenLineageRunEventBuilder.buildRun(OpenLineageRunEventBuilder.java:279)\n\tat io.openlineage.spark.agent.lifecycle.OpenLineageRunEventBuilder.buildRun(OpenLineageRunEventBuilder.java:222)\n\tat io.openlineage.spark.agent.lifecycle.SparkSQLExecutionContext.start(SparkSQLExecutionContext.java:72)\n\tat io.openlineage.spark.agent.OpenLineageSparkListener.lambda$sparkSQLExecStart$0(OpenLineageSparkListener.java:91)```","u":"U0625RZ7KR9","t":"1697840317.080859"},{"m":"<@U05JY6MN8MS>\nI am trying to contribute to Integration tests which is listed here as \nthe mentions that i can trigger CI for integration tests from forked branch.\n.\nbut i am unable to do so, is there a way to trigger CI from forked brach or do i have to get permission from someone to run the CI?\n\ni am getting this error when i run this command `sudo git-push-fork-to-upstream-branch upstream savannavalgi:hacktober`\n> ```Username for '': savannavalgi\n> Password for '': \n> remote: Permission to OpenLineage/OpenLineage.git denied to savannavalgi.\n> fatal: unable to access '': The requested URL returned error: 403```\ni have tried to configure ssh key\nalso tried to trigger CI from another brach,\nand tried all of this after fetching the latest upstream\n\ncc: <@U05JBHLPY8K> <@U01RA9B5GG2> <@U05HD9G5T17>","u":"U05KCF3EEUR","t":"1697805105.047909"},{"m":"Hey all - we've been noticing that some events go unreported by openlineage (spark) when the `AsyncEventQueue` fills up and starts dropping events. Wondering if anyone has experienced this before, and knows why it is happening? We've expanded the event queue capacity and thrown more hardware at the problem but no dice\n\nAlso as a note, the query plans from this job are pretty big - could the listener just be choking up? Happy to open a github issue as well if we suspect that it could be the listener itself having issues","u":"U03D8K119LJ","t":"1697742042.953399"},{"m":"Hello All, I am completely new for Openlineage, I have to setup the lab to conduct POC on various aspects like Lineage, metadata management , etc. As per openlineage site, i tried downloading Ubuntu, docker and binary files for Marquez. But I am lost somewhere and unable to configure whole setup. Can someone please assist in steps to start from scratch so that i can delve into the Openlineage capabilities. Many thanks","u":"U0616K9TSTZ","t":"1697597823.663129"},{"m":"Hello All :wave:!\nWe are currently trying to work the the *spark integration for OpenLineage in our Databricks instance*. The general setup is done and working with a few hicups here and there.\nBut one thing we are still struggling is how to link all spark jobs events with a Databricks job or a notebook run.\nWe´ve recently noticed that some of the events produced by OL have the \"environment-properties\" attribute with information (for our context) regarding notebook path (if it is a notebook run), or the the job run ID (if its a databricks job run). But the thing is that _these attributes are not always present._\nI ran some samples yesterday for a job with 4 notebook tasks. From all 20 json payload sent by the OL listener, only 3 presented the \"environment-properties\" attribute. Its not only happening with Databricks jobs. When i run single notebooks and each cell has its onw set of spark jobs, not all json events presented that property either.\n\nSo my question is *what is the criteria to have this attributes present or not in the event json file*? Or maybe this in an issue? <@U05T8BJD4DU> did you find out anything about this?\n\n:gear: Spark 3.4 / OL-Spark 1.4.1","u":"U05TU0U224A","t":"1697527077.180169"},{"m":"Hi team, I am running the following pyspark code in a cell:\n```print(\"SELECTING 100 RECORDS FROM METADATA TABLE\")\ndf = spark.sql(\"\"\"select * from
limit 100\"\"\")\n\nprint(\"WRITING (1) 100 RECORDS FROM METADATA TABLE\")\ndf.write.mode(\"overwrite\").format('delta').save(\"\")\ndf.createOrReplaceTempView(\"temp_metadata\")\n\nprint(\"WRITING (2) 100 RECORDS FROM METADATA TABLE\")\ndf.write.mode(\"overwrite\").format(\"delta\").save(\"\")\n\nprint(\"READING (1) 100 RECORDS FROM METADATA TABLE\")\ndf_read = spark.read.format('delta').load(\"\")\ndf_read.createOrReplaceTempView(\"metadata_1\")\n\nprint(\"DOING THE MERGE INTO SQL STEP!\")\ndf_new = spark.sql(\"\"\"\n MERGE INTO metadata_1\n USING
\n ON metadata_1.id = temp_metadata.id\n WHEN MATCHED THEN UPDATE SET \n metadata_1.id = temp_metadata.id,\n metadata_1.aspect = temp_metadata.aspect\n WHEN NOT MATCHED THEN INSERT (id, aspect) \n VALUES (temp_metadata.id, temp_metadata.aspect)\n\"\"\")```\nI am running with debug log levels. I actually don't see any of the events being logged for `SaveIntoDataSourceCommand` or the `MergeIntoCommand`, but OL is in fact emitting events to the backend. It seems like the events are just not being logged... I actually observe this for all delta table related spark sql queries...","u":"U04EZ2LPDV4","t":"1697179720.032079"},{"m":"This might be a dumb question, I guess I need to setup local Spark in order for the Spark tests to run successfully?","u":"U05T8BJD4DU","t":"1697137714.503349"},{"m":"\nFriendly reminder: this month’s TSC meeting, open to all, is tomorrow at 10 am PT: ","u":"U02LXF3HUN7","t":"1697043601.182719"},{"m":"Hi @there, I am trying to make API call to get column-lineage information could you please let me know the url construct to retrieve the same? As per the API documentation I am passing the following url to GET column-lineage: but getting error code:400. Thanks","u":"U05HK41VCH1","t":"1697040264.029839"},{"m":" When i am running this sql as part of a databricks notebook, i am recieving an OL event where i see only an output dataset and there is no input dataset or a symlink facet inside the dataset to map it to the underlying azure storage object. Can anyone kindly help on this\n```spark.sql(f\"CREATE TABLE IF NOT EXISTS covid_research.uscoviddata USING delta LOCATION ''\")\n{\n \"eventTime\": \"2023-10-11T10:47:36.296Z\",\n \"producer\": \"\",\n \"schemaURL\": \"\",\n \"eventType\": \"COMPLETE\",\n \"run\": {\n \"runId\": \"d0f40be9-b921-4c84-ac9f-f14a86c29ff7\",\n \"facets\": {\n \"spark.logicalPlan\": {\n \"_producer\": \"\",\n \"_schemaURL\": \"\",\n \"plan\": [\n {\n \"class\": \"org.apache.spark.sql.catalyst.plans.logical.CreateTable\",\n \"num-children\": 1,\n \"name\": 0,\n \"tableSchema\": [],\n \"partitioning\": [],\n \"tableSpec\": null,\n \"ignoreIfExists\": true\n },\n {\n \"class\": \"org.apache.spark.sql.catalyst.analysis.ResolvedIdentifier\",\n \"num-children\": 0,\n \"catalog\": null,\n \"identifier\": null\n }\n ]\n },\n \"spark_version\": {\n \"_producer\": \"\",\n \"_schemaURL\": \"\",\n \"spark-version\": \"3.3.0\",\n \"openlineage-spark-version\": \"1.2.2\"\n },\n \"processing_engine\": {\n \"_producer\": \"\",\n \"_schemaURL\": \"\",\n \"version\": \"3.3.0\",\n \"name\": \"spark\",\n \"openlineageAdapterVersion\": \"1.2.2\"\n }\n }\n },\n \"job\": {\n \"namespace\": \"default\",\n \"name\": \"adb-3942203504488904.4.azuredatabricks.net.create_table.covid_research_db_uscoviddata\",\n \"facets\": {}\n },\n \"inputs\": [],\n \"outputs\": [\n {\n \"namespace\": \"dbfs\",\n \"name\": \"/user/hive/warehouse/covid_research.db/uscoviddata\",\n \"facets\": {\n \"dataSource\": {\n \"_producer\": \"\",\n \"_schemaURL\": \"\",\n \"name\": \"dbfs\",\n \"uri\": \"dbfs\"\n },\n \"schema\": {\n \"_producer\": \"\",\n \"_schemaURL\": \"\",\n \"fields\": []\n },\n \"storage\": {\n \"_producer\": \"\",\n \"_schemaURL\": \"\",\n \"storageLayer\": \"unity\",\n \"fileFormat\": \"parquet\"\n },\n \"symlinks\": {\n \"_producer\": \"\",\n \"_schemaURL\": \"\",\n \"identifiers\": [\n {\n \"namespace\": \"/user/hive/warehouse/covid_research.db\",\n \"name\": \"covid_research.uscoviddata\",\n \"type\": \"TABLE\"\n }\n ]\n },\n \"lifecycleStateChange\": {\n \"_producer\": \"\",\n \"_schemaURL\": \"\",\n \"lifecycleStateChange\": \"CREATE\"\n }\n },\n \"outputFacets\": {}\n }\n ]\n}```","u":"U05QL7LN2GH","t":"1697021758.073929"},{"m":" i am trying out the databricks spark integration and in one of the events i am getting a openlineage event where the output dataset is having a facet called `symlinks` , the statement that generated this event is this sql\n```CREATE TABLE IF NOT EXISTS covid_research.covid_data \nUSING CSV\nLOCATION '' \nOPTIONS (header \"true\", inferSchema \"true\");```\nCan someone kindly let me know what this `symlinks` facet is. i tried seeing the spec but did not get it completely","u":"U05QL7LN2GH","t":"1696995819.546399"},{"m":"example:\n\n```{\"environment-properties\":{\"spark.databricks.clusterUsageTags.clusterName\":\"'s Cluster\",\"spark.databricks.job.runId\":\"\",\"spark.databricks.job.type\":\"\",\"spark.databricks.clusterUsageTags.azureSubscriptionId\":\"a4f54399-8db8-4849-adcc-a42aed1fb97f\",\"spark.databricks.notebook.path\":\"/Repos/jason.yip@tredence.com/segmentation/01_Data Prep\",\"spark.databricks.clusterUsageTags.clusterOwnerOrgId\":\"4679476628690204\",\"MountPoints\":[{\"MountPoint\":\"/databricks-datasets\",\"Source\":\"databricks-datasets\"},{\"MountPoint\":\"/Volumes\",\"Source\":\"UnityCatalogVolumes\"},{\"MountPoint\":\"/databricks/mlflow-tracking\",\"Source\":\"databricks/mlflow-tracking\"},{\"MountPoint\":\"/databricks-results\",\"Source\":\"databricks-results\"},{\"MountPoint\":\"/databricks/mlflow-registry\",\"Source\":\"databricks/mlflow-registry\"},{\"MountPoint\":\"/Volume\",\"Source\":\"DbfsReserved\"},{\"MountPoint\":\"/volumes\",\"Source\":\"DbfsReserved\"},{\"MountPoint\":\"/\",\"Source\":\"DatabricksRoot\"},{\"MountPoint\":\"/volume\",\"Source\":\"DbfsReserved\"}],\"User\":\"\",\"UserId\":\"4768657035718622\",\"OrgId\":\"4679476628690204\"}}```","u":"U05T8BJD4DU","t":"1696985639.868119"},{"m":"Any idea why \"environment-properties\" is gone in Spark 3.4+ in StartEvent?","u":"U05T8BJD4DU","t":"1696914311.793789"},{"m":"Hello. I am getting started with OL and Marquez with dbt. I am using dbt-ol. The namespace of the dataset showing up in Marquez is not the namespace I provide using OPENLINEAGE_NAMESPACE. It happens to be the same as the source in Marquez which is the snowflake account uri. It's obviously picking up the other env variable OPENLINEAGE_URL so i am pretty sure its not the environment. Is this expected?","u":"U021QJMRP47","t":"1696884935.692409"},{"m":"\n*We released OpenLineage 1.4.1!*\n*Additions:*\n• *Client:* *allow setting client’s endpoint via environment variable* <@U01HVNU6A4C> \n• *Flink: expand Iceberg source types* <@U05QA2D1XNV> \n• *Spark: add debug facet* <@U02MK6YNAQ5> \n• *Spark: enable Nessie REST catalog* \nThanks to all the contributors, especially new contributors <@U05QA2D1XNV> and !\n*Release:* \n*Changelog:* \n*Commit history:* \n*Maven:* \n*PyPI:* ","u":"U02LXF3HUN7","t":"1696879514.895109"},{"m":" I am trying out the openlineage integration of spark on databricks. There is no event getting emitted from Openlineage, I see logs saying OpenLineage Event Skipped. I am attaching the Notebook that i am trying to run and the cluster logs. Kindly can someone help me on this","u":"U05QL7LN2GH","t":"1696823976.297949"},{"m":"<@U02LXF3HUN7> can we cut a new release to include this change?\n• ","u":"U01HVNU6A4C","t":"1696591141.778179"},{"m":"The Marquez meetup in San Francisco is happening right now!\n","u":"U01DCLP0GU9","t":"1696552840.350759"},{"m":"I have created a ticket to make this easier to find. Once I get more feedback I’ll turn it into a md file in the repo: \n","u":"U01DCLP0GU9","t":"1696541652.452819"},{"m":"**\nThis month’s TSC meeting is next Thursday the 12th at 10am PT. On the tentative agenda:\n• announcements\n• recent releases\n• Airflow Summit recap\n• tutorial: migrating to the Airflow Provider\n• discussion topic: observability for OpenLineage/Marquez\n• open discussion\n• more (TBA)\nMore info and the meeting link can be found on the . All are welcome! Do you have a discussion topic, use case or integration you’d like to demo? DM me to be added to the agenda.","u":"U02LXF3HUN7","t":"1696531454.431629"},{"m":"I have cleaned up the registry proposal.\n\nIn particular:\n• I clarified that option 2 is preferred at this point.\n• I moved discussion notes to the bottom. they will go away at some point\n• Once it is stable, I’ll create a with the preferred option.\n• we need a good proposal for the core facets prefix. My suggestion is to move core facets to `core` in the registry. The drawback is prefix would be inconsistent.\n","u":"U01DCLP0GU9","t":"1696379615.265919"},{"m":"Hey everyone - does anyone have a good mechanism for alerting on issues with open lineage? For example, maybe alerting when an event times out - perhaps to prometheus or some other kind of generic endpoint? Not sure the best approach here (if the meta inf extension would be able to achieve it)","u":"U03D8K119LJ","t":"1696350897.139129"},{"m":"\n*We released OpenLineage 1.3.1!*\n*Added:*\n• Airflow: add some basic stats to the Airflow integration `#1845` \n• Airflow: add columns as schema facet for `airflow.lineage.Table` (if defined) `#2138` \n• DBT: add SQLSERVER to supported dbt profile types `#2136` \n• Spark: support for latest 3.5 `#2118` \n*Fixed:*\n• Airflow: fix find-links path in tox `#2139` \n• Airflow: add more graceful logging when no OpenLineage provider installed `#2141` \n• Spark: fix bug in PathUtils’ `prepareDatasetIdentifierFromDefaultTablePath` (CatalogTable) to correctly preserve scheme from `CatalogTable`’s location `#2142` \nThanks to all the contributors, including new contributor <@U05TZE47F2S>!\n*Release:* \n*Changelog:* \n*Commit history:* \n*Maven:* \n*PyPI:* ","u":"U02LXF3HUN7","t":"1696344963.496819"},{"m":"Hi folks - I'm wondering if its just me, but does `io.openlineage:openlineage-sql-java:1.2.2` ship with the `arm64.dylib` binary? When i try and run code that uses the Java package on an Apple M1, the binary isn't found, The workaround is to checkout 1.2.2 and then build and publish it locally.","u":"U05FLJE4GDU","t":"1696319076.770719"},{"m":"\nThe September issue of is here! This issue covers the big news about OpenLineage coming out of Airflow Summit, progress on the Airflow Provider, highlights from our meetup in Toronto, and much more.\nTo get the newsletter directly in your inbox each month, sign up .","u":"U02LXF3HUN7","t":"1696264108.497989"},{"m":"\nHello all, I’d like to open a vote to release OpenLineage 1.3.0, including:\n• support for Spark 3.5 in the Spark integration\n• scheme preservation bug fix in the Spark integration\n• find-links path in tox bug in the Airflow integration fix\n• more graceful logging when no OL provider is installed in the Airflow integration\n• columns as schema facet for airflow.lineage.Table addition\n• SQLSERVER to supported dbt profile types addition\nThree +1s from committers will authorize. Thanks in advance.","u":"U02LXF3HUN7","t":"1696262312.791719"},{"m":"Are you located in the Brussels area or within commutable distance? Interested in attending a meetup between October 16-20? If so, please DM <@U0323HG8C8H> or myself. TIA","u":"U02LXF3HUN7","t":"1695932184.205159"},{"m":"Hello community\nFirst time poster - bear with me :)\n\nI am looking to make a minor PR on the airflow integration (fixing github #2130), and the code change is easy enough, but I fail to install the python environment. I have tried the simple ones\n`OpenLineage/integration/airflow > pip install -e .`\n or\n`OpenLineage/integration/airflow > pip install -r dev-requirements.txt`\nbut they both fail on\n`ERROR: No matching distribution found for openlineage-sql==1.3.0`\n\n(which I think is an unreleased version in the git project)\n\nHow would I go about to install the requirements?\n\n//Erik\n\nPS. Sorry for posting this in general if there is a specific integration or contribution channel - I didnt find a better channel","u":"U05TZE47F2S","t":"1695883240.832669"},{"m":"Hi folks, am I correct in my observations that the Spark integration does not generate inputs and outputs for Kafka-to-Kafka pipelines?\n\n*EDIT:* Removed the crazy wall of text. Relevant GitHub issue is .","u":"U05FLJE4GDU","t":"1695831785.042079"},{"m":"*Meetup recap: Toronto Meetup @ Airflow Summit, September 18, 2023*\nIt was great to see so many members of our community at this event! I counted 32 total attendees, with all but a handful being first-timers.\n*Topics included:*\n• Presentation on the history, architecture and roadmap of the project by <@U01DCLP0GU9> and <@U01HNKK4XAM> \n• Discussion of OpenLineage support in by <@U01DCMDFHBK> \n• Presentation by *Ye Liu* and *Ivan Perepelitca* from , the social platform for data, about their integration\n• Presentation by <@U02MK6YNAQ5> about the Spark integration\n• Presentation by <@U01RA9B5GG2> about the Apache Airflow Provider\nThanks to all the presenters and attendees with a shout out to <@U01HNKK4XAM> for the help with organizing and day-of logistics, <@U02S6F54MAB> for the help with set up/clean up, and <@U0323HG8C8H> for the crucial assist with the signup sheet.\nThis was our first meetup in Toronto, and we learned some valuable lessons about planning events in new cities — the first and foremost being to ask for a pic of the building! :slightly_smiling_face: But it seemed like folks were undeterred, and the space itself lived up to expectations.\nFor a recording and clips from the meetup, head over to our .\n*Upcoming events:*\n• October 5th in San Francisco: Marquez Meetup @ Astronomer (sign up )\n• November: Warsaw meetup (details, date TBA)\n• January: London meetup (details, date TBA)\nAre you interested in hosting or co-hosting an OpenLineage or Marquez meetup? DM me!","u":"U02LXF3HUN7","t":"1695827956.140429"},{"m":" In Airflow Integration we send across a lineage Event for Dag start and complete, but that is not the case with spark integration…we don’t receive any event for the application start and complete in spark…is this expected behaviour or am i missing something?","u":"U05QL7LN2GH","t":"1695703890.171789"},{"m":"I'm using the Spark OpenLineage integration. In the `outputStatistics` output dataset facet we receive `rowCount` and `size`.\nThe Job performs a SQL insert into a MySQL table and I'm receiving the `size` as 0.\n```{\n \"outputStatistics\":\n {\n \"_producer\": \"\",\n \"_schemaURL\": \"\",\n \"rowCount\": 1,\n \"size\": 0\n }\n}```\nI'm not sure what the size means here. Does this mean number of bytes inserted/updated?\nAlso, do we have any documentation for Spark specific Job and Run facets?","u":"U05A1D80QKF","t":"1695663385.834539"},{"m":" I'm presently addressing a particular scenario that pertains to Openlineage authentication, specifically involving the use of an access key and secret.\n\nI've implemented a custom token provider called AccessKeySecretKeyTokenProvider, which extends the TokenProvider class. This token provider communicates with another service, obtaining a token and an expiration time based on the provided access key, secret, and client ID.\n\nMy goal is to retain this token in a cache prior to its expiration, thereby eliminating the need for network calls to the third-party service. Is it possible without relying on an external caching system.","u":"U05SMTVPPL3","t":"1695633110.066819"},{"m":"I am attaching the log4j, there is no openlineagecontext","u":"U05T8BJD4DU","t":"1695352570.560639"},{"m":"I installed 1.2.2 on Databricks, followed the below init script: \n\nmy cluster config looks like this:\n\nspark.openlineage.version v1\nspark.openlineage.namespace adb-5445974573286168.8#default\nspark.openlineage.endpoint v1/lineage\nspark.openlineage.url.param.code 8kZl0bo2TJfnbpFxBv-R2v7xBDj-PgWMol3yUm5iP1vaAzFu9kIZGg==\nspark.openlineage.url \n\nBut it is not calling the API, it works fine with 0.18 version","u":"U05T8BJD4DU","t":"1695347501.889769"},{"m":"I am using this accelerator that leverages OpenLineage on Databricks to publish lineage info to Purview, but it's using a rather old version of OpenLineage aka 0.18, anybody has tried it on a newer version of OpenLineage? I am facing some issues with the inputs and outputs for the same object is having different json\n","u":"U05T8BJD4DU","t":"1695335777.852519"},{"m":"Hi, we're using Custom Operators in airflow(2.5) and are planning to expose lineage via default extractors: \n*Question*: Now if we upgrade our Airflow version to 2.7 in the future, would our code be backward compatible?\nSince OpenLineage has now moved inside airflow and I think there is no concept of extractors in the latest version.","u":"U05A1D80QKF","t":"1695276670.439269"},{"m":"\nWe released OpenLineage 1.2.2!\nAdded\n• Spark: publish the `ProcessingEngineRunFacet` as part of the normal operation of the `OpenLineageSparkEventListener` `#2089` \n• Spark: capture and emit `spark.databricks.clusterUsageTags.clusterAllTags` variable from databricks environment `#2099` \nFixed\n• Common: support parsing dbt_project.yml without target-path `#2106` \n• Proxy: fix Proxy chart `#2091` \n• Python: fix serde filtering `#2044` \n• Python: use non-deprecated `apiKey` if loading it from env variables `@2029` \n• Spark: Improve RDDs on S3 integration. `#2039` \n• Flink: prevent sending `running` events after job completes `#2075` \n• Spark & Flink: Unify dataset naming from URI objects `#2083` \n• Spark: Databricks improvements `#2076` \nRemoved\n• SQL: remove sqlparser dependency from iface-java and iface-py `#2090` \nThanks to all the contributors, including new contributors <@U055N2GRT4P>, , and !\n*Release:* \n*Changelog:* \n*Commit history:* \n*Maven:* \n*PyPI:* ","u":"U02LXF3HUN7","t":"1695244138.650089"},{"m":"congrats folks :partying_face: ","u":"U05HFGKEYVB","t":"1695217014.549799"},{"m":"Hi I need help in extracting OpenLineage for PostgresOperator in json format.\nany suggestions or comments would be greatly appreciated","u":"U05SQGH8DV4","t":"1695106067.665469"},{"m":"Hi! I'm in need of help with wrapping my head around OpenLineage. My team have the goal of collecting metadata from the Airflow operators GreatExpectationsOperator, PythonOperator, MsSqlOperator and BashOperator (for dbt). Where can I see the sourcecode for what is collected for each operator, and is there support for these in the new provider *apache-airflow-providers-openlineage*? I am super confused and feel lost in the docs. :exploding_head: We are using MSSQL/ODBC to connect to our db, and this data does not seem to appear as datasets in Marquez, do I need to configure this? If so, HOW and WHERE? :smiling_face_with_tear:\n\nHappy for any help, big or small! :pray:","u":"U05K8F1T887","t":"1695039754.591479"},{"m":"It doesn't seem like there's a way to override the OL endpoint from the default (`/api/v1/lineage`) in Airflow? I tried setting the `OPENLINEAGE_ENDPOINT` environment to no avail. Based on , it seems that only `OPENLINEAGE_URL` was used to construct `HttpConfig` ?","u":"U01HVNU6A4C","t":"1694956061.909169"},{"m":" is there a way by which we could add custom headers to openlineage client in airflow, i see that provision is there for spark integration via these properties --> abcdef","u":"U05QL7LN2GH","t":"1694907627.974239"},{"m":" we have dataproc operator getting called from a dag which submits a spark job, we wanted to maintain that continuity of parent job in the spark job so according to the documentation we can acheive that by using a macro called lineage_run_id that requires task and task_instance as the parameters. The problem we are facing is that our client’s have 1000's of dags, so asking them to change this everywhere it is used is not feasible, so we thought of using the task_policy feature in the airflow…but the problem is that task_policy gives you access to only the task/operator, but we don’t have the access to the task instance..that is required as a parameter to the lineage_run_id function. Can anyone kindly help us on how should we go about this one\n```t1 = DataProcPySparkOperator(\n task_id=job_name,\n #required pyspark configuration,\n job_name=job_name,\n dataproc_pyspark_properties={\n 'spark.driver.extraJavaOptions':\n f\"-javaagent:{jar}={os.environ.get('OPENLINEAGE_URL')}/api/v1/namespaces/{os.getenv('OPENLINEAGE_NAMESPACE', 'default')}/jobs/{job_name}/runs/{{{{macros.OpenLineagePlugin.lineage_run_id(task, task_instance)}}}}?api_key={os.environ.get('OPENLINEAGE_API_KEY')}\"\n dag=dag)```","u":"U05QL7LN2GH","t":"1694849427.228709"},{"m":"\nFriendly reminder: the next OpenLineage meetup, our first in Toronto, is happening this coming Monday at 5 PM ET ","u":"U02LXF3HUN7","t":"1694793807.376729"},{"m":"Per discussion in the OpenLineage sync today here is a very early strawman proposal for an OpenLineage registry that producers and consumers could be registered in.\nFeedback or alternate proposals welcome\n\nOnce this is sufficiently fleshed out, I’ll create an actual proposal on github","u":"U01DCLP0GU9","t":"1694737381.437569"},{"m":"Hey everyone,\nAny chance we could have a *openlineage-integration-common* 1.1.1 release with the following changes..?\n• \n• ","u":"U055N2GRT4P","t":"1694700221.242579"},{"m":"Context:\n\nWe use Spark with YARN, running on Hadoop 2.x (I can't remember the exact minor version) with Hive support.\n\nProblem:\n\nI'm noticed that `CreateDataSourceAsSelectCommand` objects are _always_ transformed to an `OutputDataset` with a _namespace_ value set to `file` - which is curious, because the inputs always have a (correct) namespace of `hdfs://<name-node>` - is this a known issue? A flaw with Apache Spark? A bug in the resolution logic?\n\nFor reference:\n\n```public class CreateDataSourceTableCommandVisitor\n extends QueryPlanVisitor<CreateDataSourceTableCommand, OpenLineage.OutputDataset> {\n\n public CreateDataSourceTableCommandVisitor(OpenLineageContext context) {\n super(context);\n }\n\n @Override\n public List<OpenLineage.OutputDataset> apply(LogicalPlan x) {\n CreateDataSourceTableCommand command = (CreateDataSourceTableCommand) x;\n CatalogTable catalogTable = command.table();\n\n return Collections.singletonList(\n outputDataset()\n .getDataset(\n PathUtils.fromCatalogTable(catalogTable),\n catalogTable.schema(),\n OpenLineage.LifecycleStateChangeDatasetFacet.LifecycleStateChange.CREATE));\n }\n}```\nRunning this: `cat events.log | jq '{eventTime: .eventTime, eventType: .eventType, runId: .run.runId, jobNamespace: .job.namespace, jobName: .job.name, outputs: .outputs[] | {namespace: .namespace, name: .name}, inputs: .inputs[] | {namespace: .namespace, name: .name}}'`\n\nThis is an output:\n```{\n \"eventTime\": \"2023-09-13T16:01:27.059Z\",\n \"eventType\": \"START\",\n \"runId\": \"bbbb5763-3615-46c0-95ca-1fc398c91d5d\",\n \"jobNamespace\": \"spark.cluster-1\",\n \"jobName\": \"ol_hadoop_test.execute_create_data_source_table_as_select_command.dhawes_db_ol_test_hadoop_tgt\",\n \"outputs\": {\n \"namespace\": \"file\",\n \"name\": \"/user/hive/warehouse/dhawes.db/ol_test_hadoop_tgt\"\n },\n \"inputs\": {\n \"namespace\": \"\",\n \"name\": \"/user/hive/warehouse/dhawes.db/ol_test_hadoop_src\"\n }\n}```","u":"U05FLJE4GDU","t":"1694686815.337029"},{"m":"\nThis month’s TSC meeting, open to all, is tomorrow: ","u":"U02LXF3HUN7","t":"1694629232.934029"},{"m":"I am exploring Spark - OpenLineage integration (using the latest PySpark and OL versions). I tested a simple pipeline which:\n• Reads JSON data into PySpark DataFrame\n• Apply data transformations\n• Write transformed data to MySQL database\nObserved that we receive 4 events (2 `START` and 2 `COMPLETE`) for the same job name. The events are almost identical with a small diff in the facets. All the events share the same `runId`, and we don't get any `parentRunId`.\nTeam, can you please confirm if this behaviour is expected? Seems to be different from the Airflow integration where we relate jobs to Parent Jobs.","u":"U05A1D80QKF","t":"1694583867.900909"},{"m":" has anyone succeded in getting a custom extractor to work in GCP Cloud Composer or AWS MWAA, seems like there is no way","u":"U05QL7LN2GH","t":"1694553961.188719"},{"m":"I am trying to run Google Cloud Composer where i have added the openlineage-airflow pypi packagae as a dependency and have added the env OPENLINEAGE_EXTRACTORS to point to my custom extractor. I have added a folder by name dependencies and inside that i have placed my extractor file, and the path given to OPENLINEAGE_EXTRACTORS is dependencies.<file_name>.<extractor_class_name>…still it fails with the exception saying No module named ‘dependencies’. Can anyone kindly help me out on correcting my mistake","u":"U05QL7LN2GH","t":"1694545905.974339"},{"m":"This particular code in docker-compose exits with code 1 because it is unable to find wait-for-it.sh, file in the container. I have checked the mounting path from the local machine, It is correct and the path on the container for Marquez is also correct i.e. /usr/src/app but it is unable to mount the wait-for-it.sh. Does anyone know why is this? This code exists in the open lineage repository as well \n```# Marquez as an OpenLineage Client\n api:\n image: marquezproject/marquez\n container_name: marquez-api\n ports:\n - \"5000:5000\"\n - \"5001:5001\"\n volumes:\n - ./docker/wait-for-it.sh:/usr/src/app/wait-for-it.sh\n links:\n - \"db:postgres\"\n depends_on:\n - db\n entrypoint: [ \"./wait-for-it.sh\", \"db:5432\", \"--\", \"./entrypoint.sh\" ]```","u":"U05QNRSQW1E","t":"1694520846.519609"},{"m":"Opened a PR for this here: ","u":"U04AZ7992SU","t":"1694468262.274069"},{"m":"I’m seeing some odd behavior with my http transport when upgrading airflow/openlineage-airflow from 2.3.2 -> 2.6.3 and 0.24.0 -> 0.28.0. Previously I had a config like this that let me provide my own auth tokens. However, after upgrading I’m getting a 401 from the endpoint and further debugging seems to reveal that we’re not using the token provided in my TokenProvider. Does anyone know if something changed between these versions that could be causing this? (more details in :thread: )\n```transport:\n type: http\n url: \n auth:\n type: some.fully.qualified.classpath```","u":"U04AZ7992SU","t":"1694466446.599329"},{"m":"\nThe first Toronto OpenLineage Meetup, featuring a presentation by recent adopter , is just one week away. On the agenda:\n1. *Evolution of spec presentation/discussion (project background/history)*\n2. *State of the community*\n3. *Integrating OpenLineage with (by special guests & )*\n4. *Spark/Column lineage update*\n5. *Airflow Provider update*\n6. *Roadmap Discussion*\n*Find more details and RSVP *.","u":"U02LXF3HUN7","t":"1694441261.486759"},{"m":"\nThis month’s TSC meeting is next Thursday the 14th at 10am PT. On the tentative agenda:\n• announcements\n• recent releases\n• demo: Spark integration tests in Databricks runtime\n• open discussion\n• more (TBA)\nMore info and the meeting link can be found on the . All are welcome! Also, feel free to reply or DM me with discussion topics, agenda items, etc.","u":"U02LXF3HUN7","t":"1694113940.400549"},{"m":"Has there been any conversation on the extensibility of facets/concepts? E.g.:\n• how does one extends the list of run states to add a paused/resumed state?\n• how does one extend to add a created at field?","u":"U0595Q78HUG","t":"1694036652.124299"},{"m":"Hello Everyone,\n\nI've been diving into the Marquez codebase and found a performance bottleneck in `JobDao.java` for the query related to `namespaceName=MyNameSpace` `limit=10` and 12s with `limit=25`. I managed to optimize it using CTEs, and the execution times dropped dramatically to 300ms (for `limit=100`) and under 100ms (for `limit=25` ) on the same cluster.\nIssue link : \n\nI believe there's even more room for optimization, especially if we adjust the `job_facets_view` to include the `namespace_name` column.\n\nWould the team be open to a PR where I share the optimized query and discuss potential further refinements? I believe these changes could significantly enhance the Marquez web UI experience.\n\nPR link : \n\nLooking forward to your feedback.","u":"U05HBLE7YPL","t":"1694032987.624809"},{"m":"it looks like my dynamic task mapping in Airflow has the same run ID in marquez, so even if I am processing 100 files, there is only one version of the data. is there a way to have a separate version of each dynamic task so I can track the filename etc?","u":"U05NMJ0NBUK","t":"1693877705.781699"},{"m":"Also, another small clarification is that when using `MergeIntoCommand`, I'm receiving the lineage events on the backend, but I cannot seem to find any logging of the payload when I enable debug mode in openlineage. I remember there was a similar issue reported by another user in the past. May I check if it might be possible to help with this? It's making debugging quite hard for these cases. Thanks!","u":"U04EZ2LPDV4","t":"1693823945.734419"},{"m":"Hi guys, I'd like to capture the `spark.databricks.clusterUsageTags.clusterAllTags` property from databricks. However, the value of this is a list of keys, and therefore cannot be supported by custom environment facet builder.\nI was thinking that capturing this property might be useful for most databricks workloads, and whether it might make sense to auto-capture it along with other databricks variables, similar to how we capture mount points for the databricks jobs.\nDoes this sound okay? If so, then I can help to contribute this functionality","u":"U04EZ2LPDV4","t":"1693813108.356499"},{"m":"\nThe is out now! Please to get it directly in your inbox each month.","u":"U02LXF3HUN7","t":"1693602981.025489"},{"m":"It sounds like there have been a few announcements at Google Next:\n\n","u":"U01DCLP0GU9","t":"1693519820.292119"},{"m":"Will the August meeting be put up at soon? (usually it’s up in a few days :slightly_smiling_face:","u":"U0323HG8C8H","t":"1693510399.153829"},{"m":"ok,it`s my first use thie lineage tool. first,I added dependences in my pom.xml like this:\n<dependency>\n <groupId>io.openlineage</groupId>\n <artifactId>openlineage-java</artifactId>\n <version>0.12.0</version>\n </dependency>\n <dependency>\n <groupId>org.apache.logging.log4j</groupId>\n <artifactId>log4j-api</artifactId>\n <version>2.7</version>\n </dependency>\n <dependency>\n <groupId>org.apache.logging.log4j</groupId>\n <artifactId>log4j-core</artifactId>\n <version>2.7</version>\n </dependency>\n <dependency>\n <groupId>org.apache.logging.log4j</groupId>\n <artifactId>log4j-slf4j-impl</artifactId>\n <version>2.7</version>\n </dependency>\n <dependency>\n <groupId>io.openlineage</groupId>\n <artifactId>openlineage-spark</artifactId>\n <version>0.30.1</version>\n </dependency>\n\nmy spark version is 3.3.1 and the version can not change\n\nsecond, in file Openlineage/intergration/spark I enter command : docker-compose up and follow the steps in this doc:\n\nthere is no erro when i use notebook to execute pyspark for openlineage and I could get json message.\nbut after I enter \"docker-compose up\" ,I want to use my Idea tool to execute scala code like above,the erro happend like above. It seems that I does not configure the environment correctly. so how can i fix the problem .","u":"U05NGJ8AM8X","t":"1693468312.450209"},{"m":"hello,everyone,i can run openLineage spark code in my notebook with python,but when use my idea to execute scala code like this:\nimport org.apache.spark.internal.Logging\nimport org.apache.spark.sql.SparkSession\nimport io.openlineage.client.OpenLineageClientUtils.loadOpenLineageYaml\nimport org.apache.spark.scheduler.{SparkListener, SparkListenerApplicationEnd, SparkListenerApplicationStart}\nimport sun.java2d.marlin.MarlinUtils.logInfo\nobject Test {\n def main(args: Array[String]): Unit = {\n\n val spark = SparkSession\n .builder()\n .master(\"local\")\n .appName(\"test\")\n .config(\"spark.jars.packages\",\"io.openlineage:openlineage-spark:0.12.0\")\n .config(\"spark.extraListeners\",\"io.openlineage.spark.agent.OpenLineageSparkListener\")\n .config(\"spark.openlineage.transport.type\",\"console\")\n .getOrCreate()\n\n spark.sparkContext.setLogLevel(\"INFO\")\n\n //spark.sparkContext.addSparkListener(new MySparkAppListener)\n import spark.implicits._\n val input = Seq((1, \"zs\", 2020), (2, \"ls\", 2023)).toDF(\"id\", \"name\", \"year\")\n\n input.select(\"id\", \"name\").orderBy(\"id\").show()\n\n }\n\n}\n\nthere is something wrong:\nException in thread \"spark-listener-group-shared\" java.lang.NoSuchMethodError: io.openlineage.client.OpenLineageClientUtils.loadOpenLineageYaml(Ljava/io/InputStream;)Lio/openlineage/client/OpenLineageYaml;\n\tat io.openlineage.spark.agent.ArgumentParser.extractOpenlineageConfFromSparkConf(ArgumentParser.java:114)\n\tat io.openlineage.spark.agent.ArgumentParser.parse(ArgumentParser.java:78)\n\tat io.openlineage.spark.agent.OpenLineageSparkListener.initializeContextFactoryIfNotInitialized(OpenLineageSparkListener.java:277)\n\tat io.openlineage.spark.agent.OpenLineageSparkListener.onApplicationStart(OpenLineageSparkListener.java:267)\n\tat org.apache.spark.scheduler.SparkListenerBus.doPostEvent(SparkListenerBus.scala:55)\n\tat org.apache.spark.scheduler.SparkListenerBus.doPostEvent$(SparkListenerBus.scala:28)\n\tat org.apache.spark.scheduler.AsyncEventQueue.doPostEvent(AsyncEventQueue.scala:37)\n\tat org.apache.spark.scheduler.AsyncEventQueue.doPostEvent(AsyncEventQueue.scala:37)\n\tat org.apache.spark.util.ListenerBus.postToAll(ListenerBus.scala:117)\n\tat org.apache.spark.util.ListenerBus.postToAll$(ListenerBus.scala:101)\n\tat org.apache.spark.scheduler.AsyncEventQueue.super$postToAll(AsyncEventQueue.scala:105)\n\tat org.apache.spark.scheduler.AsyncEventQueue.$anonfun$dispatch$1(AsyncEventQueue.scala:105)\n\tat scala.runtime.java8.JFunction0$mcJ$sp.apply(JFunction0$mcJ$sp.java:23)\n\tat scala.util.DynamicVariable.withValue(DynamicVariable.scala:62)\n\tat $apache$spark$scheduler$AsyncEventQueue$$dispatch(AsyncEventQueue.scala:100)\n\tat org.apache.spark.scheduler.AsyncEventQueue$$anon$2.$anonfun$run$1(AsyncEventQueue.scala:96)\n\tat org.apache.spark.util.Utils$.tryOrStopSparkContext(Utils.scala:1446)\n\tat org.apache.spark.scheduler.AsyncEventQueue$$anon$2.run(AsyncEventQueue.scala:96)\n\ni want to know how can i set idea scala environment correctly","u":"U05NGJ8AM8X","t":"1693463508.522729"},{"m":"Can anyone let 3 people stuck downstairs into the 7th floor?","u":"U05EC8WB74N","t":"1693445911.744069"},{"m":"\nFriendly reminder: there’s a meetup at Astronomer’s offices in SF!","u":"U02LXF3HUN7","t":"1693410605.894959"},{"m":"Hi, Will really appreciate if someone can guide me or provide me any pointer - if they have been able to implement authentication/authorization for access to Marquez. Have not seen much info around it. Any pointers greatly appreciated. Thanks in advance.","u":"U05JY6MN8MS","t":"1693397729.978309"},{"m":"for namespaces, if my data is moving between sources (SFTP -> GCS -> Azure Blob (synapse connects to parquet datasets) then should my namespace be based on the client I am working with? my current namespace has been to refer to the bucket, but that falls apart when considering the data sources and some destinations. perhaps I should just add a field for client-name instead to have a consolidated view?","u":"U05NMJ0NBUK","t":"1693329152.193929"},{"m":"hi folks, for now I'm producing `.jsonl` (or `.ndjson` ) files with one event per line, do you know if there's any way to validate those? would standard JSON Schema tools work?","u":"U05HFGKEYVB","t":"1693300839.710459"},{"m":"Hello, I'm currently in the process of following the instructions outlined in the provided getting started guide at . However, I've encountered a problem while attempting to complete **Step 1** of the guide. Unfortunately, I'm encountering an internal server error at this stage. I did manage to successfully run Marquez, but it appears that there might be an issue that needs to be addressed. I have attached screen shots.","u":"U05QNRSQW1E","t":"1693293484.701439"},{"m":"New on the OpenLineage blog: , including:\n• the critical improvements it brings to the integration\n• the high-level design\n• implementation details\n• an example operator\n• planned enhancements\n• a list of supported operators\n• more.\nThe post, by <@U01RA9B5GG2>, <@U01DCLP0GU9> and myself is live now on the OpenLineage blog.","u":"U02LXF3HUN7","t":"1693267537.810959"},{"m":"\nThe agenda for the on 9/18 has been updated. This promises to be an exciting, richly productive discussion. Don’t miss it if you’ll be in the area!\n1. Intros\n2. Evolution of spec presentation/discussion (project background/history)\n3. State of the community\n4. Spark/Column lineage update\n5. Airflow Provider update \n6. Roadmap Discussion\n7. Action items review/next steps\n","u":"U02LXF3HUN7","t":"1693258111.112809"},{"m":"Hi folks. I have some pure golang jobs from which I need to emit OL events to Marquez. Is the right way to go about this to generate a Golang client from the Marquez OpenAPI spec and use that client from my go jobs?","u":"U05PVS8GRJ6","t":"1693243558.640159"},{"m":"and something else: I understand that Marquez does not yet support the 2.0 spec, hence it's incompatible with static metadata right? I tried to emit a list of `DatasetEvent` s and got `HTTPError: 422 Client Error: Unprocessable Entity for url: ` (I'm using a `FileTransport` for now)","u":"U05HFGKEYVB","t":"1693212561.369509"},{"m":"hi folks, I'm looking into exporting static metadata, and found that `DatasetEvent` requires a `eventTime`, which in my mind doesn't make sense for static events. I'm setting it to `None` and the Python client seems to work, but wanted to ask if I'm missing something.","u":"U05HFGKEYVB","t":"1693212473.810659"},{"m":"hi Openlineage team , we would like to join one of your meetups(me and <@U05HK41VCH1> nad @Phil Rolph and we're wondering if you are hosting any meetups after the 18/9 ? We are trying to join this but air - tickets are quite expensive","u":"U05J5GRKY10","t":"1692975450.380969"},{"m":"\nFriendly reminder: our next in-person meetup is next Wednesday, August 30th in San Francisco at Astronomer’s offices in the Financial District. You can sign up and find the details on the .","u":"U02LXF3HUN7","t":"1692973763.570629"},{"m":"\nWe released OpenLineage 1.1.0, including:\nAdditions:\n• Flink: create Openlineage configuration based on Flink configuration `#2033` \n• Java: add Javadocs to the Java client `#2004` \n• Spark: append output dataset name to a job name `#2036` \n• Spark: support Spark 3.4.1 `#2057` \nFixes:\n• Flink: fix a bug when getting schema for `KafkaSink` `#2042` \n• Spark: fix ignored event `adaptive_spark_plan` in Databricks `#2061` \nPlus additional bug fixes, doc changes and more.\nThanks to all the contributors, especially new contributors @pentium3 and <@U05HBLE7YPL>!\n*Release:* \n*Changelog:* \n*Commit history:* \n*Maven:* \n*PyPI:* ","u":"U02LXF3HUN7","t":"1692817450.338859"},{"m":"Hey folks! Do we have clear step-by-step documentation on how we can leverage the `ServiceLoader` based approach for injecting specific OpenLineage customisations for tweaking the transport type with defaults / tweaking column level lineage etc?","u":"U05JBHLPY8K","t":"1692810528.463669"},{"m":"Approve a new release please :slightly_smiling_face:\n• Fix spark integration filtering Databricks events. ","u":"U05HBLE7YPL","t":"1692802510.386629"}],"C01NAFMBVEY":[],"C030F1J0264":[],"C04E3Q18RR9":[],"C04JPTTC876":[],"C04QSV0GG23":[],"C04THH1V90X":[],"C051C93UZK9":[],"C055GGUFMHQ":[],"C056YHEU680":[{"m":"<@U05TQPZ4R4L> has joined the channel","u":"U05TQPZ4R4L","t":"1695502057.851259"},{"m":"Some pictures from last night","u":"U01DCLP0GU9","t":"1693538290.446329"},{"m":"<@U05Q3HT6PBR> has joined the channel","u":"U05Q3HT6PBR","t":"1693520941.967439"},{"m":"Time: 5:30-8:30 pm\nAddress: 8 California St., San Francisco, CA, seventh floor\nGetting in: someone from Astronomer will be in the lobby to direct you","u":"U02LXF3HUN7","t":"1693422775.587509"},{"m":"Adding the venue info in case it’s more convenient than the meetup page:","u":"U02LXF3HUN7","t":"1693422678.406409"}],"C05N442RQUA":[{"m":"Hi, if you’re wondering if you’re in the right place: look for Uncle Tetsu’s Cheesecake nextdoor and for the address (600 Bay St) above the door. The building is an older one (unlike the meeting space itself, which is modern and well-appointed)","u":"U02LXF3HUN7","t":"1695068433.208409"},{"m":"Looking forward to seeing you on Monday! Here’s the time/place info again for your convenience:\n• Date: 9/18\n• Time: 5-8:00 PM ET\n• Place: Canarts, 600 Bay St., #410 (around the corner from the Airflow Summit venue)\n• Venue phone: \n• Meetup page with more info and signup: \nPlease send a message if you find yourself stuck in the lobby, etc.","u":"U02LXF3HUN7","t":"1694794649.751439"},{"m":"<@U05SXDWVA7K> has joined the channel","u":"U05SXDWVA7K","t":"1694787071.646859"},{"m":"\nIt’s hard to believe this is happening in just one week! Here’s the updated agenda:\n1. *Intros*\n2. *Evolution of spec presentation/discussion (project background/history)*\n3. *State of the community*\n4. *Integrating OpenLineage with (by special guests & )*\n5. *Spark/Column lineage update*\n6. *Airflow Provider update*\n7. *Roadmap Discussion*\n8. *Action items review/next steps*\nFind the details and RSVP .","u":"U02LXF3HUN7","t":"1694441637.116609"},{"m":"Most OpenLineage regular contributors will be there. It will be fun to be all in person. Everyone is encouraged to join","u":"U01DCLP0GU9","t":"1693624251.155569"},{"m":"really looking forward to meeting all of you in Toronto!!","u":"U01HNKK4XAM","t":"1692984822.264569"},{"m":"Some belated updates on this in case you’re not aware:\n• Date: 9/18\n• Time: 5-8:00 PM ET\n• Place: Canarts, 600 Bay St., #410 (around the corner from the Airflow Summit venue)\n• Venue phone: \n• Meetup for more info and to sign up: ","u":"U02LXF3HUN7","t":"1692984607.290939"}],"C05PD7VJ52S":[{"m":"<@U05QHG1NJ8J> has joined the channel","u":"U05QHG1NJ8J","t":"1693512257.995209"},{"m":"<@U01RA9B5GG2> has joined the channel","u":"U01RA9B5GG2","t":"1692984861.392349"},{"m":"Yes, hope so! Thank you for your interest in joining a meetup!","u":"U02LXF3HUN7","t":"1692983998.092189"},{"m":"hopefully meet you soon in London","u":"U05HK41VCH1","t":"1692982281.255129"},{"m":"Thanks Michael for starting this channel","u":"U05HK41VCH1","t":"1692982268.668049"},{"m":"yes absolutely will give you an answer by Monday","u":"U05J5GRKY10","t":"1692979169.558489"},{"m":"OK! Would you please let me know when you know, and we’ll go from there?","u":"U02LXF3HUN7","t":"1692979126.425069"},{"m":"and if that not the case i can provide personal space","u":"U05J5GRKY10","t":"1692979080.273399"},{"m":"I am pretty sure you can use our 6point6 offices or at least part of it","u":"U05J5GRKY10","t":"1692979067.390839"},{"m":"I will have to confirm but 99% yes","u":"U05J5GRKY10","t":"1692978958.692039"},{"m":"Great! Do you happen to have space we could use?","u":"U02LXF3HUN7","t":"1692978871.220909"},{"m":"thats perfect !","u":"U05J5GRKY10","t":"1692978852.583609"},{"m":"Hi George, nice to meet you. Thanks for asking about future meetups. Would November be too soon, or what’s a good timeframe for you all?","u":"U02LXF3HUN7","t":"1692978834.508549"},{"m":"thanks so much !","u":"U05J5GRKY10","t":"1692978816.416879"},{"m":"Hi Michael","u":"U05J5GRKY10","t":"1692978769.711289"},{"m":"<@U05HK41VCH1> has joined the channel","u":"U05HK41VCH1","t":"1692978760.556819"},{"m":"<@U01HNKK4XAM> has joined the channel","u":"U01HNKK4XAM","t":"1692978735.461499"},{"m":"<@U05J5GRKY10> has joined the channel","u":"U05J5GRKY10","t":"1692978735.367849"},{"m":"<@U02LXF3HUN7> has joined the channel","u":"U02LXF3HUN7","t":"1692978725.076179"}],"C05U3UC85LM":[{"m":"<@U0620HU51HA> has joined the channel","u":"U0620HU51HA","t":"1697734944.171989"},{"m":"<@U05U9K21LSG> would be great if we could get your eyes on this PR: ","u":"U01HNKK4XAM","t":"1697201275.793679"},{"m":"Just seeing this, we had a company holiday yesterday. Yes, fluent data sources are our new way of connecting to data and the older \"block-style\" is deprecated and will be removed when we cut 0.18.0. I'm not sure of the timing of that but likely in the next couple months.","u":"U05U9K21LSG","t":"1696979522.397239"},{"m":"<@U05U9929K3N> <@U05U9K21LSG> ^^","u":"U01HNKK4XAM","t":"1696860879.713919"},{"m":"Hello guys! I’ve been looking recently into changes in GX.\n\nis this the major change you’d like to introduce in OL<-> GX?","u":"U02S6F54MAB","t":"1696852012.119669"},{"m":"<@U05U9K21LSG> has joined the channel","u":"U05U9K21LSG","t":"1695916771.601439"},{"m":"<@U05U9929K3N> it was great meeting earlier, looking forward to collaborating on this!","u":"U01HNKK4XAM","t":"1695840258.594299"},{"m":"<@U01RA9B5GG2> has joined the channel","u":"U01RA9B5GG2","t":"1695836477.653589"},{"m":"<@U02S6F54MAB> has joined the channel","u":"U02S6F54MAB","t":"1695836303.953919"},{"m":"<@U01HNKK4XAM> has joined the channel","u":"U01HNKK4XAM","t":"1695836303.827639"},{"m":"<@U05U9929K3N> has joined the channel","u":"U05U9929K3N","t":"1695836303.727049"},{"m":"<@U02LXF3HUN7> has joined the channel","u":"U02LXF3HUN7","t":"1695836283.617309"}],"C065PQ4TL8K":[{"m":"Maybe move today's meeting earlier, since no one from west coast is joining? <@U01HNKK4XAM>","u":"U01RA9B5GG2","t":"1700562211.366219"},{"m":"I’m off on vacation. See you in a week","u":"U01DCLP0GU9","t":"1700272614.735719"},{"m":"just searching for OpenLineage in the Datahub code base. They have an “interesting” approach? ","u":"U01DCLP0GU9","t":"1700246539.228259"},{"m":"CFP for Berlin Buzzwords went up: \nStill over 3 months to submit :slightly_smiling_face:","u":"U01RA9B5GG2","t":"1700155042.082759"},{"m":"worlds are colliding: 6point6 has been acquired by Accenture","u":"U02LXF3HUN7","t":"1700145084.414099"},{"m":"Any opinions about a free task management alternative to the free version of Notion (10-person limit)? Looking at Trello for keeping track of talks.","u":"U02LXF3HUN7","t":"1700088623.669029"},{"m":"have we discussed adding column level lineage support to Airflow? ","u":"U01DCMDFHBK","t":"1700087546.032789"},{"m":"Apparently an admin can view a Slack archive at any time at this URL: . Only public channels are available, though.","u":"U02LXF3HUN7","t":"1700078359.877599"},{"m":"Anyone have thoughts about how to address the question about “pain points” here? . (Listing pros is easy — it’s the cons we don’t have boilerplate for)","u":"U02LXF3HUN7","t":"1700078230.775579"},{"m":"is it time to *support hudi*?","u":"U02S6F54MAB","t":"1700068651.517579"},{"m":"Got the doc + poc for hook-level coverage: ","u":"U01RA9B5GG2","t":"1700066684.350639"},{"m":"hey look, more fun\n","u":"U02S6F54MAB","t":"1700040937.040239"},{"m":"also, what about this PR? ","u":"U01DCMDFHBK","t":"1700037370.235629"},{"m":"`_Minor_: We can consider defining a _run_state column and eventually dropping the event_type. That is, we can consider columns prefixed with _ to be \"remappings\" of OL properties to Marquez.` -> didn't get this one. Is it for now or some future plans?","u":"U02MK6YNAQ5","t":"1700037282.474539"},{"m":"<@U02MK6YNAQ5> approved PR with minor comments, I think the is one comment we’ll need to address before merging; otherwise solid work dude :ok_hand:","u":"U01DCMDFHBK","t":"1700037147.106879"},{"m":"\nfun PR incoming","u":"U02S6F54MAB","t":"1700004648.584649"},{"m":":wave:","u":"U01RA9B5GG2","t":"1699987988.642529"},{"m":":ocean:","u":"U053LLVTHRN","t":"1699982987.125549"},{"m":":wave:","u":"U01DCMDFHBK","t":"1699982333.485469"},{"m":":wave: ","u":"U01DCLP0GU9","t":"1699982322.990799"},{"m":":wave:","u":"U02LXF3HUN7","t":"1699982179.651129"},{"m":"<@U053LLVTHRN> has joined the channel","u":"U053LLVTHRN","t":"1699982042.414079"},{"m":":wave:","u":"U02S6F54MAB","t":"1699982037.267109"},{"m":"<@U05KKM07PJP> has joined the channel","u":"U05KKM07PJP","t":"1699982026.646699"},{"m":"<@U01DCMDFHBK> has joined the channel","u":"U01DCMDFHBK","t":"1699982026.554329"},{"m":"<@U02LXF3HUN7> has joined the channel","u":"U02LXF3HUN7","t":"1699982026.462039"},{"m":"<@U02S6F54MAB> has joined the channel","u":"U02S6F54MAB","t":"1699982026.350149"},{"m":"<@U02MK6YNAQ5> has joined the channel","u":"U02MK6YNAQ5","t":"1699982026.266479"},{"m":"<@U01DCLP0GU9> has joined the channel","u":"U01DCLP0GU9","t":"1699982026.191829"},{"m":"<@U01RA9B5GG2> has joined the channel","u":"U01RA9B5GG2","t":"1699981990.850589"},{"m":"<@U01HNKK4XAM> has joined the channel","u":"U01HNKK4XAM","t":"1699981986.459199"}]},"pages":{"C01CK9T7HKR":["1692802510.386629","1692802510.386629"],"C056YHEU680":["1693422678.406409","1693422678.406409"],"C05N442RQUA":["1692984607.290939","1692984607.290939"],"C05PD7VJ52S":["1692978725.076179","1692978725.076179"],"C05U3UC85LM":["1695836283.617309","1695836283.617309"],"C065PQ4TL8K":["1699981986.459199","1699981986.459199"]}}; \ No newline at end of file diff --git a/slack-archive/data/slack-archive.json b/slack-archive/data/slack-archive.json new file mode 100644 index 0000000..a0b334e --- /dev/null +++ b/slack-archive/data/slack-archive.json @@ -0,0 +1,76 @@ +{ + "channels": { + "C01CK9T7HKR": { + "messages": 118 + }, + "C01NAFMBVEY": { + "fullyDownloaded": true, + "messages": 0 + }, + "C030F1J0264": { + "messages": 0 + }, + "C04E3Q18RR9": { + "messages": 0 + }, + "C04JPTTC876": { + "messages": 0 + }, + "C04QSV0GG23": { + "fullyDownloaded": true, + "messages": 0 + }, + "C04THH1V90X": { + "messages": 0 + }, + "C051C93UZK9": { + "messages": 0 + }, + "C055GGUFMHQ": { + "messages": 0 + }, + "C056YHEU680": { + "messages": 5 + }, + "C05N442RQUA": { + "messages": 7 + }, + "C05PD7VJ52S": { + "messages": 19 + }, + "C05U3UC85LM": { + "messages": 12 + }, + "C065PQ4TL8K": { + "messages": 31 + } + }, + "auth": { + "ok": true, + "url": "https://openlineage.slack.com/", + "team": "OpenLineage", + "user": "michael282", + "team_id": "T01CWUYP5AR", + "user_id": "U02LXF3HUN7", + "is_enterprise_install": false, + "response_metadata": { + "scopes": [ + "read", + "client", + "identify", + "post", + "channels:history", + "groups:history", + "im:history", + "mpim:history", + "channels:read", + "files:read", + "groups:read", + "im:read", + "mpim:read", + "users:read", + "remote_files:read" + ] + } + } +} \ No newline at end of file diff --git a/slack-archive/data/users.json b/slack-archive/data/users.json new file mode 100644 index 0000000..1cc5e37 --- /dev/null +++ b/slack-archive/data/users.json @@ -0,0 +1,2918 @@ +{ + "U066S97A90C": { + "id": "U066S97A90C", + "team_id": "T01CWUYP5AR", + "name": "rwojcik", + "deleted": false, + "color": "2b6836", + "real_name": "Rafał Wójcik", + "tz": "Europe/Warsaw", + "tz_label": "Central European Time", + "tz_offset": 3600, + "profile": { + "title": "", + "phone": "", + "skype": "", + "real_name": "Rafał Wójcik", + "real_name_normalized": "Rafal Wojcik", + "display_name": "Rafał Wójcik", + "display_name_normalized": "Rafal Wojcik", + "fields": {}, + "status_text": "", + "status_emoji": "", + "status_emoji_display_info": [], + "status_expiration": 0, + "avatar_hash": "09ba2780a8ae", + "image_original": "https://avatars.slack-edge.com/2023-11-21/6227762962771_09ba2780a8aeaf7658c6_original.png", + "is_custom_image": true, + "email": "rwojcik@griddynamics.com", + "first_name": "Rafał", + "last_name": "Wójcik", + "image_24": "https://avatars.slack-edge.com/2023-11-21/6227762962771_09ba2780a8aeaf7658c6_24.png", + "image_32": "https://avatars.slack-edge.com/2023-11-21/6227762962771_09ba2780a8aeaf7658c6_32.png", + "image_48": "https://avatars.slack-edge.com/2023-11-21/6227762962771_09ba2780a8aeaf7658c6_48.png", + "image_72": "https://avatars.slack-edge.com/2023-11-21/6227762962771_09ba2780a8aeaf7658c6_72.png", + "image_192": "https://avatars.slack-edge.com/2023-11-21/6227762962771_09ba2780a8aeaf7658c6_192.png", + "image_512": "https://avatars.slack-edge.com/2023-11-21/6227762962771_09ba2780a8aeaf7658c6_512.png", + "image_1024": "https://avatars.slack-edge.com/2023-11-21/6227762962771_09ba2780a8aeaf7658c6_1024.png", + "status_text_canonical": "", + "team": "T01CWUYP5AR" + }, + "is_admin": false, + "is_owner": false, + "is_primary_owner": false, + "is_restricted": false, + "is_ultra_restricted": false, + "is_bot": false, + "is_app_user": false, + "updated": 1700567815, + "is_email_confirmed": true, + "has_2fa": false, + "who_can_share_contact_card": "EVERYONE" + }, + "U066CNW85D3": { + "id": "U066CNW85D3", + "team_id": "T01CWUYP5AR", + "name": "karthik.nandagiri", + "deleted": false, + "color": "684b6c", + "real_name": "karthik nandagiri", + "tz": "Asia/Kolkata", + "tz_label": "India Standard Time", + "tz_offset": 19800, + "profile": { + "title": "", + "phone": "", + "skype": "", + "real_name": "karthik nandagiri", + "real_name_normalized": "karthik nandagiri", + "display_name": "karthik nandagiri", + "display_name_normalized": "karthik nandagiri", + "fields": {}, + "status_text": "", + "status_emoji": "", + "status_emoji_display_info": [], + "status_expiration": 0, + "avatar_hash": "e8d0a2a05e71", + "image_original": "https://avatars.slack-edge.com/2023-11-19/6205176178887_e8d0a2a05e71471d1c78_original.jpg", + "is_custom_image": true, + "email": "karthik.nandagiri@gmail.com", + "first_name": "karthik", + "last_name": "nandagiri", + "image_24": "https://avatars.slack-edge.com/2023-11-19/6205176178887_e8d0a2a05e71471d1c78_24.jpg", + "image_32": "https://avatars.slack-edge.com/2023-11-19/6205176178887_e8d0a2a05e71471d1c78_32.jpg", + "image_48": "https://avatars.slack-edge.com/2023-11-19/6205176178887_e8d0a2a05e71471d1c78_48.jpg", + "image_72": "https://avatars.slack-edge.com/2023-11-19/6205176178887_e8d0a2a05e71471d1c78_72.jpg", + "image_192": "https://avatars.slack-edge.com/2023-11-19/6205176178887_e8d0a2a05e71471d1c78_192.jpg", + "image_512": "https://avatars.slack-edge.com/2023-11-19/6205176178887_e8d0a2a05e71471d1c78_512.jpg", + "image_1024": "https://avatars.slack-edge.com/2023-11-19/6205176178887_e8d0a2a05e71471d1c78_1024.jpg", + "status_text_canonical": "", + "team": "T01CWUYP5AR" + }, + "is_admin": false, + "is_owner": false, + "is_primary_owner": false, + "is_restricted": false, + "is_ultra_restricted": false, + "is_bot": false, + "is_app_user": false, + "updated": 1700456127, + "is_email_confirmed": true, + "has_2fa": false, + "who_can_share_contact_card": "EVERYONE" + }, + "U066HKFCHUG": { + "id": "U066HKFCHUG", + "team_id": "T01CWUYP5AR", + "name": "naresh.naresh36", + "deleted": false, + "color": "73769d", + "real_name": "Naresh reddy", + "tz": "Asia/Kolkata", + "tz_label": "India Standard Time", + "tz_offset": 19800, + "profile": { + "title": "", + "phone": "", + "skype": "", + "real_name": "Naresh reddy", + "real_name_normalized": "Naresh reddy", + "display_name": "Naresh reddy", + "display_name_normalized": "Naresh reddy", + "fields": {}, + "status_text": "", + "status_emoji": "", + "status_emoji_display_info": [], + "status_expiration": 0, + "avatar_hash": "91e664fce7c3", + "image_original": "https://avatars.slack-edge.com/2023-11-15/6192035069510_91e664fce7c3faeee1e7_original.jpg", + "is_custom_image": true, + "email": "naresh.naresh36@gmail.com", + "first_name": "Naresh", + "last_name": "reddy", + "image_24": "https://avatars.slack-edge.com/2023-11-15/6192035069510_91e664fce7c3faeee1e7_24.jpg", + "image_32": "https://avatars.slack-edge.com/2023-11-15/6192035069510_91e664fce7c3faeee1e7_32.jpg", + "image_48": "https://avatars.slack-edge.com/2023-11-15/6192035069510_91e664fce7c3faeee1e7_48.jpg", + "image_72": "https://avatars.slack-edge.com/2023-11-15/6192035069510_91e664fce7c3faeee1e7_72.jpg", + "image_192": "https://avatars.slack-edge.com/2023-11-15/6192035069510_91e664fce7c3faeee1e7_192.jpg", + "image_512": "https://avatars.slack-edge.com/2023-11-15/6192035069510_91e664fce7c3faeee1e7_512.jpg", + "image_1024": "https://avatars.slack-edge.com/2023-11-15/6192035069510_91e664fce7c3faeee1e7_1024.jpg", + "status_text_canonical": "", + "team": "T01CWUYP5AR" + }, + "is_admin": false, + "is_owner": false, + "is_primary_owner": false, + "is_restricted": false, + "is_ultra_restricted": false, + "is_bot": false, + "is_app_user": false, + "updated": 1700049396, + "is_email_confirmed": true, + "has_2fa": false, + "who_can_share_contact_card": "EVERYONE" + }, + "U05T8BJD4DU": { + "id": "U05T8BJD4DU", + "team_id": "T01CWUYP5AR", + "name": "jasonyip", + "deleted": false, + "color": "e96699", + "real_name": "Jason Yip", + "tz": "America/Los_Angeles", + "tz_label": "Pacific Standard Time", + "tz_offset": -28800, + "profile": { + "title": "", + "phone": "", + "skype": "", + "real_name": "Jason Yip", + "real_name_normalized": "Jason Yip", + "display_name": "", + "display_name_normalized": "", + "fields": {}, + "status_text": "", + "status_emoji": "", + "status_emoji_display_info": [], + "status_expiration": 0, + "avatar_hash": "gec3e494f453", + "email": "jasonyip@gmail.com", + "first_name": "Jason", + "last_name": "Yip", + "image_24": "https://secure.gravatar.com/avatar/ec3e494f453bc030f8732cda718b8ac5.jpg?s=24&d=https%3A%2F%2Fa.slack-edge.com%2Fdf10d%2Fimg%2Favatars%2Fava_0025-24.png", + "image_32": "https://secure.gravatar.com/avatar/ec3e494f453bc030f8732cda718b8ac5.jpg?s=32&d=https%3A%2F%2Fa.slack-edge.com%2Fdf10d%2Fimg%2Favatars%2Fava_0025-32.png", + "image_48": "https://secure.gravatar.com/avatar/ec3e494f453bc030f8732cda718b8ac5.jpg?s=48&d=https%3A%2F%2Fa.slack-edge.com%2Fdf10d%2Fimg%2Favatars%2Fava_0025-48.png", + "image_72": "https://secure.gravatar.com/avatar/ec3e494f453bc030f8732cda718b8ac5.jpg?s=72&d=https%3A%2F%2Fa.slack-edge.com%2Fdf10d%2Fimg%2Favatars%2Fava_0025-72.png", + "image_192": "https://secure.gravatar.com/avatar/ec3e494f453bc030f8732cda718b8ac5.jpg?s=192&d=https%3A%2F%2Fa.slack-edge.com%2Fdf10d%2Fimg%2Favatars%2Fava_0025-192.png", + "image_512": "https://secure.gravatar.com/avatar/ec3e494f453bc030f8732cda718b8ac5.jpg?s=512&d=https%3A%2F%2Fa.slack-edge.com%2Fdf10d%2Fimg%2Favatars%2Fava_0025-512.png", + "status_text_canonical": "", + "team": "T01CWUYP5AR" + }, + "is_admin": false, + "is_owner": false, + "is_primary_owner": false, + "is_restricted": false, + "is_ultra_restricted": false, + "is_bot": false, + "is_app_user": false, + "updated": 1695335591, + "is_email_confirmed": true, + "has_2fa": false, + "who_can_share_contact_card": "EVERYONE" + }, + "U02LXF3HUN7": { + "id": "U02LXF3HUN7", + "team_id": "T01CWUYP5AR", + "name": "michael282", + "deleted": false, + "color": "50a0cf", + "real_name": "Michael Robinson", + "tz": "America/New_York", + "tz_label": "Eastern Standard Time", + "tz_offset": -18000, + "profile": { + "title": "", + "phone": "", + "skype": "", + "real_name": "Michael Robinson", + "real_name_normalized": "Michael Robinson", + "display_name": "", + "display_name_normalized": "", + "fields": {}, + "status_text": "", + "status_emoji": "", + "status_emoji_display_info": [], + "status_expiration": 0, + "avatar_hash": "66fea720e950", + "image_original": "https://avatars.slack-edge.com/2022-01-25/3019716733729_66fea720e9504dc08144_original.jpg", + "is_custom_image": true, + "email": "michael.robinson@astronomer.io", + "huddle_state": "default_unset", + "huddle_state_expiration_ts": 0, + "first_name": "Michael", + "last_name": "Robinson", + "image_24": "https://avatars.slack-edge.com/2022-01-25/3019716733729_66fea720e9504dc08144_24.jpg", + "image_32": "https://avatars.slack-edge.com/2022-01-25/3019716733729_66fea720e9504dc08144_32.jpg", + "image_48": "https://avatars.slack-edge.com/2022-01-25/3019716733729_66fea720e9504dc08144_48.jpg", + "image_72": "https://avatars.slack-edge.com/2022-01-25/3019716733729_66fea720e9504dc08144_72.jpg", + "image_192": "https://avatars.slack-edge.com/2022-01-25/3019716733729_66fea720e9504dc08144_192.jpg", + "image_512": "https://avatars.slack-edge.com/2022-01-25/3019716733729_66fea720e9504dc08144_512.jpg", + "image_1024": "https://avatars.slack-edge.com/2022-01-25/3019716733729_66fea720e9504dc08144_1024.jpg", + "status_text_canonical": "", + "team": "T01CWUYP5AR" + }, + "is_admin": true, + "is_owner": true, + "is_primary_owner": false, + "is_restricted": false, + "is_ultra_restricted": false, + "is_bot": false, + "is_app_user": false, + "updated": 1700183405, + "is_email_confirmed": true, + "has_2fa": false, + "who_can_share_contact_card": "EVERYONE" + }, + "U05TU0U224A": { + "id": "U05TU0U224A", + "team_id": "T01CWUYP5AR", + "name": "rodrigo.maia", + "deleted": false, + "color": "684b6c", + "real_name": "Rodrigo Maia", + "tz": "Europe/Belgrade", + "tz_label": "Central European Time", + "tz_offset": 3600, + "profile": { + "title": "", + "phone": "", + "skype": "", + "real_name": "Rodrigo Maia", + "real_name_normalized": "Rodrigo Maia", + "display_name": "Rodrigo Maia", + "display_name_normalized": "Rodrigo Maia", + "fields": {}, + "status_text": "", + "status_emoji": "", + "status_emoji_display_info": [], + "status_expiration": 0, + "avatar_hash": "1373b2f5d8ba", + "image_original": "https://avatars.slack-edge.com/2023-09-25/5943104201445_1373b2f5d8babe2b8c2a_original.png", + "is_custom_image": true, + "email": "rodrigo.maia@manta.io", + "first_name": "Rodrigo", + "last_name": "Maia", + "image_24": "https://avatars.slack-edge.com/2023-09-25/5943104201445_1373b2f5d8babe2b8c2a_24.png", + "image_32": "https://avatars.slack-edge.com/2023-09-25/5943104201445_1373b2f5d8babe2b8c2a_32.png", + "image_48": "https://avatars.slack-edge.com/2023-09-25/5943104201445_1373b2f5d8babe2b8c2a_48.png", + "image_72": "https://avatars.slack-edge.com/2023-09-25/5943104201445_1373b2f5d8babe2b8c2a_72.png", + "image_192": "https://avatars.slack-edge.com/2023-09-25/5943104201445_1373b2f5d8babe2b8c2a_192.png", + "image_512": "https://avatars.slack-edge.com/2023-09-25/5943104201445_1373b2f5d8babe2b8c2a_512.png", + "image_1024": "https://avatars.slack-edge.com/2023-09-25/5943104201445_1373b2f5d8babe2b8c2a_1024.png", + "status_text_canonical": "", + "team": "T01CWUYP5AR" + }, + "is_admin": false, + "is_owner": false, + "is_primary_owner": false, + "is_restricted": false, + "is_ultra_restricted": false, + "is_bot": false, + "is_app_user": false, + "updated": 1699862761, + "is_email_confirmed": true, + "has_2fa": false, + "who_can_share_contact_card": "EVERYONE" + }, + "U05NMJ0NBUK": { + "id": "U05NMJ0NBUK", + "team_id": "T01CWUYP5AR", + "name": "lance.dacey2", + "deleted": false, + "color": "bb86b7", + "real_name": "ldacey", + "tz": "America/Los_Angeles", + "tz_label": "Pacific Standard Time", + "tz_offset": -28800, + "profile": { + "title": "", + "phone": "", + "skype": "", + "real_name": "ldacey", + "real_name_normalized": "ldacey", + "display_name": "ldacey", + "display_name_normalized": "ldacey", + "fields": {}, + "status_text": "", + "status_emoji": "", + "status_emoji_display_info": [], + "status_expiration": 0, + "avatar_hash": "gc77c4db9ebb", + "email": "lance.dacey2@sutherlandglobal.com", + "first_name": "ldacey", + "last_name": "", + "image_24": "https://secure.gravatar.com/avatar/c77c4db9ebb850f6b5974815f40575b8.jpg?s=24&d=https%3A%2F%2Fa.slack-edge.com%2Fdf10d%2Fimg%2Favatars%2Fava_0002-24.png", + "image_32": "https://secure.gravatar.com/avatar/c77c4db9ebb850f6b5974815f40575b8.jpg?s=32&d=https%3A%2F%2Fa.slack-edge.com%2Fdf10d%2Fimg%2Favatars%2Fava_0002-32.png", + "image_48": "https://secure.gravatar.com/avatar/c77c4db9ebb850f6b5974815f40575b8.jpg?s=48&d=https%3A%2F%2Fa.slack-edge.com%2Fdf10d%2Fimg%2Favatars%2Fava_0002-48.png", + "image_72": "https://secure.gravatar.com/avatar/c77c4db9ebb850f6b5974815f40575b8.jpg?s=72&d=https%3A%2F%2Fa.slack-edge.com%2Fdf10d%2Fimg%2Favatars%2Fava_0002-72.png", + "image_192": "https://secure.gravatar.com/avatar/c77c4db9ebb850f6b5974815f40575b8.jpg?s=192&d=https%3A%2F%2Fa.slack-edge.com%2Fdf10d%2Fimg%2Favatars%2Fava_0002-192.png", + "image_512": "https://secure.gravatar.com/avatar/c77c4db9ebb850f6b5974815f40575b8.jpg?s=512&d=https%3A%2F%2Fa.slack-edge.com%2Fdf10d%2Fimg%2Favatars%2Fava_0002-512.png", + "status_text_canonical": "", + "team": "T01CWUYP5AR" + }, + "is_admin": false, + "is_owner": false, + "is_primary_owner": false, + "is_restricted": false, + "is_ultra_restricted": false, + "is_bot": false, + "is_app_user": false, + "updated": 1696099450, + "is_email_confirmed": true, + "has_2fa": false, + "who_can_share_contact_card": "EVERYONE" + }, + "U05J9LZ355L": { + "id": "U05J9LZ355L", + "team_id": "T01CWUYP5AR", + "name": "yannick.libert.partne", + "deleted": false, + "color": "e23f99", + "real_name": "Yannick Libert", + "tz": "Europe/Brussels", + "tz_label": "Central European Time", + "tz_offset": 3600, + "profile": { + "title": "Tech Lead @ Decathlon", + "phone": "", + "skype": "", + "real_name": "Yannick Libert", + "real_name_normalized": "Yannick Libert", + "display_name": "Yannick Libert", + "display_name_normalized": "Yannick Libert", + "fields": { + "Xf03UL3S8CP8": { + "value": "Tech Lead @ Decathlon", + "alt": "" + } + }, + "status_text": "", + "status_emoji": "", + "status_emoji_display_info": [], + "status_expiration": 0, + "avatar_hash": "2d7ad30e24c5", + "image_original": "https://avatars.slack-edge.com/2023-07-24/5613838546967_2d7ad30e24c5e74bfdbb_original.png", + "is_custom_image": true, + "email": "yannick.libert.partner@decathlon.com", + "first_name": "Yannick", + "last_name": "Libert", + "image_24": "https://avatars.slack-edge.com/2023-07-24/5613838546967_2d7ad30e24c5e74bfdbb_24.png", + "image_32": "https://avatars.slack-edge.com/2023-07-24/5613838546967_2d7ad30e24c5e74bfdbb_32.png", + "image_48": "https://avatars.slack-edge.com/2023-07-24/5613838546967_2d7ad30e24c5e74bfdbb_48.png", + "image_72": "https://avatars.slack-edge.com/2023-07-24/5613838546967_2d7ad30e24c5e74bfdbb_72.png", + "image_192": "https://avatars.slack-edge.com/2023-07-24/5613838546967_2d7ad30e24c5e74bfdbb_192.png", + "image_512": "https://avatars.slack-edge.com/2023-07-24/5613838546967_2d7ad30e24c5e74bfdbb_512.png", + "image_1024": "https://avatars.slack-edge.com/2023-07-24/5613838546967_2d7ad30e24c5e74bfdbb_1024.png", + "status_text_canonical": "", + "team": "T01CWUYP5AR" + }, + "is_admin": false, + "is_owner": false, + "is_primary_owner": false, + "is_restricted": false, + "is_ultra_restricted": false, + "is_bot": false, + "is_app_user": false, + "updated": 1693830902, + "is_email_confirmed": true, + "has_2fa": false, + "who_can_share_contact_card": "EVERYONE" + }, + "U05JBHLPY8K": { + "id": "U05JBHLPY8K", + "team_id": "T01CWUYP5AR", + "name": "athityakumar", + "deleted": false, + "color": "84b22f", + "real_name": "Athitya Kumar", + "tz": "Asia/Kolkata", + "tz_label": "India Standard Time", + "tz_offset": 19800, + "profile": { + "title": "", + "phone": "", + "skype": "", + "real_name": "Athitya Kumar", + "real_name_normalized": "Athitya Kumar", + "display_name": "Athitya Kumar", + "display_name_normalized": "Athitya Kumar", + "fields": {}, + "status_text": "", + "status_emoji": "", + "status_emoji_display_info": [], + "status_expiration": 0, + "avatar_hash": "055f51bd1a05", + "image_original": "https://avatars.slack-edge.com/2023-07-23/5623598828611_055f51bd1a0583028e25_original.jpg", + "is_custom_image": true, + "email": "athityakumar@gmail.com", + "first_name": "Athitya", + "last_name": "Kumar", + "image_24": "https://avatars.slack-edge.com/2023-07-23/5623598828611_055f51bd1a0583028e25_24.jpg", + "image_32": "https://avatars.slack-edge.com/2023-07-23/5623598828611_055f51bd1a0583028e25_32.jpg", + "image_48": "https://avatars.slack-edge.com/2023-07-23/5623598828611_055f51bd1a0583028e25_48.jpg", + "image_72": "https://avatars.slack-edge.com/2023-07-23/5623598828611_055f51bd1a0583028e25_72.jpg", + "image_192": "https://avatars.slack-edge.com/2023-07-23/5623598828611_055f51bd1a0583028e25_192.jpg", + "image_512": "https://avatars.slack-edge.com/2023-07-23/5623598828611_055f51bd1a0583028e25_512.jpg", + "image_1024": "https://avatars.slack-edge.com/2023-07-23/5623598828611_055f51bd1a0583028e25_1024.jpg", + "status_text_canonical": "", + "team": "T01CWUYP5AR" + }, + "is_admin": false, + "is_owner": false, + "is_primary_owner": false, + "is_restricted": false, + "is_ultra_restricted": false, + "is_bot": false, + "is_app_user": false, + "updated": 1690133734, + "is_email_confirmed": true, + "has_2fa": false, + "who_can_share_contact_card": "EVERYONE" + }, + "U0635GK8Y14": { + "id": "U0635GK8Y14", + "team_id": "T01CWUYP5AR", + "name": "david.goss", + "deleted": false, + "color": "e475df", + "real_name": "David Goss", + "tz": "Europe/London", + "tz_label": "Greenwich Mean Time", + "tz_offset": 0, + "profile": { + "title": "", + "phone": "", + "skype": "", + "real_name": "David Goss", + "real_name_normalized": "David Goss", + "display_name": "David Goss", + "display_name_normalized": "David Goss", + "fields": {}, + "status_text": "", + "status_emoji": "", + "status_emoji_display_info": [], + "status_expiration": 0, + "avatar_hash": "18447d963fbf", + "image_original": "https://avatars.slack-edge.com/2023-10-30/6126846084449_18447d963fbf13826099_original.jpg", + "is_custom_image": true, + "email": "david.goss@matillion.com", + "first_name": "David", + "last_name": "Goss", + "image_24": "https://avatars.slack-edge.com/2023-10-30/6126846084449_18447d963fbf13826099_24.jpg", + "image_32": "https://avatars.slack-edge.com/2023-10-30/6126846084449_18447d963fbf13826099_32.jpg", + "image_48": "https://avatars.slack-edge.com/2023-10-30/6126846084449_18447d963fbf13826099_48.jpg", + "image_72": "https://avatars.slack-edge.com/2023-10-30/6126846084449_18447d963fbf13826099_72.jpg", + "image_192": "https://avatars.slack-edge.com/2023-10-30/6126846084449_18447d963fbf13826099_192.jpg", + "image_512": "https://avatars.slack-edge.com/2023-10-30/6126846084449_18447d963fbf13826099_512.jpg", + "image_1024": "https://avatars.slack-edge.com/2023-10-30/6126846084449_18447d963fbf13826099_1024.jpg", + "status_text_canonical": "", + "team": "T01CWUYP5AR" + }, + "is_admin": false, + "is_owner": false, + "is_primary_owner": false, + "is_restricted": false, + "is_ultra_restricted": false, + "is_bot": false, + "is_app_user": false, + "updated": 1698656974, + "is_email_confirmed": true, + "has_2fa": false, + "who_can_share_contact_card": "EVERYONE" + }, + "U062Q95A1FG": { + "id": "U062Q95A1FG", + "team_id": "T01CWUYP5AR", + "name": "n.priya88", + "deleted": false, + "color": "827327", + "real_name": "priya narayana", + "tz": "Asia/Kolkata", + "tz_label": "India Standard Time", + "tz_offset": 19800, + "profile": { + "title": "", + "phone": "", + "skype": "", + "real_name": "priya narayana", + "real_name_normalized": "priya narayana", + "display_name": "priya narayana", + "display_name_normalized": "priya narayana", + "fields": {}, + "status_text": "", + "status_emoji": "", + "status_emoji_display_info": [], + "status_expiration": 0, + "avatar_hash": "2017b9ef7939", + "image_original": "https://avatars.slack-edge.com/2023-10-26/6084416738247_2017b9ef79397fadc4f2_original.png", + "is_custom_image": true, + "email": "n.priya88@gmail.com", + "first_name": "priya", + "last_name": "narayana", + "image_24": "https://avatars.slack-edge.com/2023-10-26/6084416738247_2017b9ef79397fadc4f2_24.png", + "image_32": "https://avatars.slack-edge.com/2023-10-26/6084416738247_2017b9ef79397fadc4f2_32.png", + "image_48": "https://avatars.slack-edge.com/2023-10-26/6084416738247_2017b9ef79397fadc4f2_48.png", + "image_72": "https://avatars.slack-edge.com/2023-10-26/6084416738247_2017b9ef79397fadc4f2_72.png", + "image_192": "https://avatars.slack-edge.com/2023-10-26/6084416738247_2017b9ef79397fadc4f2_192.png", + "image_512": "https://avatars.slack-edge.com/2023-10-26/6084416738247_2017b9ef79397fadc4f2_512.png", + "image_1024": "https://avatars.slack-edge.com/2023-10-26/6084416738247_2017b9ef79397fadc4f2_1024.png", + "status_text_canonical": "", + "team": "T01CWUYP5AR" + }, + "is_admin": false, + "is_owner": false, + "is_primary_owner": false, + "is_restricted": false, + "is_ultra_restricted": false, + "is_bot": false, + "is_app_user": false, + "updated": 1698314037, + "is_email_confirmed": true, + "has_2fa": false, + "who_can_share_contact_card": "EVERYONE" + }, + "U063YP6UJJ0": { + "id": "U063YP6UJJ0", + "team_id": "T01CWUYP5AR", + "name": "fangmik", + "deleted": false, + "color": "dd8527", + "real_name": "Mike Fang", + "tz": "America/Los_Angeles", + "tz_label": "Pacific Standard Time", + "tz_offset": -28800, + "profile": { + "title": "", + "phone": "", + "skype": "", + "real_name": "Mike Fang", + "real_name_normalized": "Mike Fang", + "display_name": "Mike Fang", + "display_name_normalized": "Mike Fang", + "fields": {}, + "status_text": "", + "status_emoji": "", + "status_emoji_display_info": [], + "status_expiration": 0, + "avatar_hash": "ge39c36e2357", + "email": "fangmik@amazon.com", + "first_name": "Mike", + "last_name": "Fang", + "image_24": "https://secure.gravatar.com/avatar/e39c36e2357d636e6fc6fe040be50b8a.jpg?s=24&d=https%3A%2F%2Fa.slack-edge.com%2Fdf10d%2Fimg%2Favatars%2Fava_0016-24.png", + "image_32": "https://secure.gravatar.com/avatar/e39c36e2357d636e6fc6fe040be50b8a.jpg?s=32&d=https%3A%2F%2Fa.slack-edge.com%2Fdf10d%2Fimg%2Favatars%2Fava_0016-32.png", + "image_48": "https://secure.gravatar.com/avatar/e39c36e2357d636e6fc6fe040be50b8a.jpg?s=48&d=https%3A%2F%2Fa.slack-edge.com%2Fdf10d%2Fimg%2Favatars%2Fava_0016-48.png", + "image_72": "https://secure.gravatar.com/avatar/e39c36e2357d636e6fc6fe040be50b8a.jpg?s=72&d=https%3A%2F%2Fa.slack-edge.com%2Fdf10d%2Fimg%2Favatars%2Fava_0016-72.png", + "image_192": "https://secure.gravatar.com/avatar/e39c36e2357d636e6fc6fe040be50b8a.jpg?s=192&d=https%3A%2F%2Fa.slack-edge.com%2Fdf10d%2Fimg%2Favatars%2Fava_0016-192.png", + "image_512": "https://secure.gravatar.com/avatar/e39c36e2357d636e6fc6fe040be50b8a.jpg?s=512&d=https%3A%2F%2Fa.slack-edge.com%2Fdf10d%2Fimg%2Favatars%2Fava_0016-512.png", + "status_text_canonical": "", + "team": "T01CWUYP5AR" + }, + "is_admin": false, + "is_owner": false, + "is_primary_owner": false, + "is_restricted": false, + "is_ultra_restricted": false, + "is_bot": false, + "is_app_user": false, + "updated": 1698884598, + "is_email_confirmed": true, + "has_2fa": false, + "who_can_share_contact_card": "EVERYONE" + }, + "U04AZ7992SU": { + "id": "U04AZ7992SU", + "team_id": "T01CWUYP5AR", + "name": "john490", + "deleted": false, + "color": "d1707d", + "real_name": "John Lukenoff", + "tz": "America/Los_Angeles", + "tz_label": "Pacific Standard Time", + "tz_offset": -28800, + "profile": { + "title": "", + "phone": "", + "skype": "", + "real_name": "John Lukenoff", + "real_name_normalized": "John Lukenoff", + "display_name": "John Lukenoff", + "display_name_normalized": "John Lukenoff", + "fields": {}, + "status_text": "", + "status_emoji": "", + "status_emoji_display_info": [], + "status_expiration": 0, + "avatar_hash": "bbc3847f5026", + "image_original": "https://avatars.slack-edge.com/2022-11-09/4373247371600_bbc3847f5026244d5ddd_original.jpg", + "is_custom_image": true, + "email": "john@jlukenoff.com", + "first_name": "John", + "last_name": "Lukenoff", + "image_24": "https://avatars.slack-edge.com/2022-11-09/4373247371600_bbc3847f5026244d5ddd_24.jpg", + "image_32": "https://avatars.slack-edge.com/2022-11-09/4373247371600_bbc3847f5026244d5ddd_32.jpg", + "image_48": "https://avatars.slack-edge.com/2022-11-09/4373247371600_bbc3847f5026244d5ddd_48.jpg", + "image_72": "https://avatars.slack-edge.com/2022-11-09/4373247371600_bbc3847f5026244d5ddd_72.jpg", + "image_192": "https://avatars.slack-edge.com/2022-11-09/4373247371600_bbc3847f5026244d5ddd_192.jpg", + "image_512": "https://avatars.slack-edge.com/2022-11-09/4373247371600_bbc3847f5026244d5ddd_512.jpg", + "image_1024": "https://avatars.slack-edge.com/2022-11-09/4373247371600_bbc3847f5026244d5ddd_1024.jpg", + "status_text_canonical": "", + "team": "T01CWUYP5AR" + }, + "is_admin": false, + "is_owner": false, + "is_primary_owner": false, + "is_restricted": false, + "is_ultra_restricted": false, + "is_bot": false, + "is_app_user": false, + "updated": 1683517767, + "is_email_confirmed": true, + "has_2fa": false, + "who_can_share_contact_card": "EVERYONE" + }, + "U062WLFMRTP": { + "id": "U062WLFMRTP", + "team_id": "T01CWUYP5AR", + "name": "hloomba", + "deleted": false, + "color": "84b22f", + "real_name": "harsh loomba", + "tz": "America/Los_Angeles", + "tz_label": "Pacific Standard Time", + "tz_offset": -28800, + "profile": { + "title": "", + "phone": "", + "skype": "", + "real_name": "harsh loomba", + "real_name_normalized": "harsh loomba", + "display_name": "harsh loomba", + "display_name_normalized": "harsh loomba", + "fields": {}, + "status_text": "", + "status_emoji": "", + "status_emoji_display_info": [], + "status_expiration": 0, + "avatar_hash": "gd4df32d7eed", + "email": "hloomba@upgrade.com", + "first_name": "harsh", + "last_name": "loomba", + "image_24": "https://secure.gravatar.com/avatar/d4df32d7eed4ad9e92731404104c2b0a.jpg?s=24&d=https%3A%2F%2Fa.slack-edge.com%2Fdf10d%2Fimg%2Favatars%2Fava_0006-24.png", + "image_32": "https://secure.gravatar.com/avatar/d4df32d7eed4ad9e92731404104c2b0a.jpg?s=32&d=https%3A%2F%2Fa.slack-edge.com%2Fdf10d%2Fimg%2Favatars%2Fava_0006-32.png", + "image_48": "https://secure.gravatar.com/avatar/d4df32d7eed4ad9e92731404104c2b0a.jpg?s=48&d=https%3A%2F%2Fa.slack-edge.com%2Fdf10d%2Fimg%2Favatars%2Fava_0006-48.png", + "image_72": "https://secure.gravatar.com/avatar/d4df32d7eed4ad9e92731404104c2b0a.jpg?s=72&d=https%3A%2F%2Fa.slack-edge.com%2Fdf10d%2Fimg%2Favatars%2Fava_0006-72.png", + "image_192": "https://secure.gravatar.com/avatar/d4df32d7eed4ad9e92731404104c2b0a.jpg?s=192&d=https%3A%2F%2Fa.slack-edge.com%2Fdf10d%2Fimg%2Favatars%2Fava_0006-192.png", + "image_512": "https://secure.gravatar.com/avatar/d4df32d7eed4ad9e92731404104c2b0a.jpg?s=512&d=https%3A%2F%2Fa.slack-edge.com%2Fdf10d%2Fimg%2Favatars%2Fava_0006-512.png", + "status_text_canonical": "", + "team": "T01CWUYP5AR" + }, + "is_admin": false, + "is_owner": false, + "is_primary_owner": false, + "is_restricted": false, + "is_ultra_restricted": false, + "is_bot": false, + "is_app_user": false, + "updated": 1698340239, + "is_email_confirmed": true, + "has_2fa": false, + "who_can_share_contact_card": "EVERYONE" + }, + "U05CAULTYG2": { + "id": "U05CAULTYG2", + "team_id": "T01CWUYP5AR", + "name": "kkandaswamy", + "deleted": false, + "color": "de5f24", + "real_name": "Kavitha", + "tz": "America/New_York", + "tz_label": "Eastern Standard Time", + "tz_offset": -18000, + "profile": { + "title": "", + "phone": "", + "skype": "", + "real_name": "Kavitha", + "real_name_normalized": "Kavitha", + "display_name": "Kavitha", + "display_name_normalized": "Kavitha", + "fields": {}, + "status_text": "", + "status_emoji": "", + "status_emoji_display_info": [], + "status_expiration": 0, + "avatar_hash": "g359455742a1", + "email": "kkandaswamy@cardinalcommerce.com", + "first_name": "Kavitha", + "last_name": "", + "image_24": "https://secure.gravatar.com/avatar/359455742a1a5fda53ba2d6d5c0b9c69.jpg?s=24&d=https%3A%2F%2Fa.slack-edge.com%2Fdf10d%2Fimg%2Favatars%2Fava_0011-24.png", + "image_32": "https://secure.gravatar.com/avatar/359455742a1a5fda53ba2d6d5c0b9c69.jpg?s=32&d=https%3A%2F%2Fa.slack-edge.com%2Fdf10d%2Fimg%2Favatars%2Fava_0011-32.png", + "image_48": "https://secure.gravatar.com/avatar/359455742a1a5fda53ba2d6d5c0b9c69.jpg?s=48&d=https%3A%2F%2Fa.slack-edge.com%2Fdf10d%2Fimg%2Favatars%2Fava_0011-48.png", + "image_72": "https://secure.gravatar.com/avatar/359455742a1a5fda53ba2d6d5c0b9c69.jpg?s=72&d=https%3A%2F%2Fa.slack-edge.com%2Fdf10d%2Fimg%2Favatars%2Fava_0011-72.png", + "image_192": "https://secure.gravatar.com/avatar/359455742a1a5fda53ba2d6d5c0b9c69.jpg?s=192&d=https%3A%2F%2Fa.slack-edge.com%2Fdf10d%2Fimg%2Favatars%2Fava_0011-192.png", + "image_512": "https://secure.gravatar.com/avatar/359455742a1a5fda53ba2d6d5c0b9c69.jpg?s=512&d=https%3A%2F%2Fa.slack-edge.com%2Fdf10d%2Fimg%2Favatars%2Fava_0011-512.png", + "status_text_canonical": "", + "team": "T01CWUYP5AR" + }, + "is_admin": false, + "is_owner": false, + "is_primary_owner": false, + "is_restricted": false, + "is_ultra_restricted": false, + "is_bot": false, + "is_app_user": false, + "updated": 1686672269, + "is_email_confirmed": true, + "has_2fa": false, + "who_can_share_contact_card": "EVERYONE" + }, + "U06315TMT61": { + "id": "U06315TMT61", + "team_id": "T01CWUYP5AR", + "name": "splicer9904", + "deleted": false, + "color": "4ec0d6", + "real_name": "Hitesh", + "tz": "Asia/Kolkata", + "tz_label": "India Standard Time", + "tz_offset": 19800, + "profile": { + "title": "", + "phone": "", + "skype": "", + "real_name": "Hitesh", + "real_name_normalized": "Hitesh", + "display_name": "Hitesh", + "display_name_normalized": "Hitesh", + "fields": {}, + "status_text": "", + "status_emoji": "", + "status_emoji_display_info": [], + "status_expiration": 0, + "avatar_hash": "ge6f8bfec602", + "email": "splicer9904@gmail.com", + "first_name": "Hitesh", + "last_name": "", + "image_24": "https://secure.gravatar.com/avatar/e6f8bfec6021c5b6b108906268cd65ea.jpg?s=24&d=https%3A%2F%2Fa.slack-edge.com%2Fdf10d%2Fimg%2Favatars%2Fava_0001-24.png", + "image_32": "https://secure.gravatar.com/avatar/e6f8bfec6021c5b6b108906268cd65ea.jpg?s=32&d=https%3A%2F%2Fa.slack-edge.com%2Fdf10d%2Fimg%2Favatars%2Fava_0001-32.png", + "image_48": "https://secure.gravatar.com/avatar/e6f8bfec6021c5b6b108906268cd65ea.jpg?s=48&d=https%3A%2F%2Fa.slack-edge.com%2Fdf10d%2Fimg%2Favatars%2Fava_0001-48.png", + "image_72": "https://secure.gravatar.com/avatar/e6f8bfec6021c5b6b108906268cd65ea.jpg?s=72&d=https%3A%2F%2Fa.slack-edge.com%2Fdf10d%2Fimg%2Favatars%2Fava_0001-72.png", + "image_192": "https://secure.gravatar.com/avatar/e6f8bfec6021c5b6b108906268cd65ea.jpg?s=192&d=https%3A%2F%2Fa.slack-edge.com%2Fdf10d%2Fimg%2Favatars%2Fava_0001-192.png", + "image_512": "https://secure.gravatar.com/avatar/e6f8bfec6021c5b6b108906268cd65ea.jpg?s=512&d=https%3A%2F%2Fa.slack-edge.com%2Fdf10d%2Fimg%2Favatars%2Fava_0001-512.png", + "status_text_canonical": "", + "team": "T01CWUYP5AR" + }, + "is_admin": false, + "is_owner": false, + "is_primary_owner": false, + "is_restricted": false, + "is_ultra_restricted": false, + "is_bot": false, + "is_app_user": false, + "updated": 1699161704, + "is_email_confirmed": true, + "has_2fa": false, + "who_can_share_contact_card": "EVERYONE" + }, + "U0625RZ7KR9": { + "id": "U0625RZ7KR9", + "team_id": "T01CWUYP5AR", + "name": "kpraveen420", + "deleted": false, + "color": "43761b", + "real_name": "praveen kanamarlapudi", + "tz": "America/Los_Angeles", + "tz_label": "Pacific Standard Time", + "tz_offset": -28800, + "profile": { + "title": "", + "phone": "", + "skype": "", + "real_name": "praveen kanamarlapudi", + "real_name_normalized": "praveen kanamarlapudi", + "display_name": "praveen kanamarlapudi", + "display_name_normalized": "praveen kanamarlapudi", + "fields": {}, + "status_text": "", + "status_emoji": "", + "status_emoji_display_info": [], + "status_expiration": 0, + "avatar_hash": "ac6b9c556b2a", + "image_original": "https://avatars.slack-edge.com/2023-10-20/6067337928646_ac6b9c556b2a0741b284_original.jpg", + "is_custom_image": true, + "email": "kpraveen420@gmail.com", + "first_name": "praveen", + "last_name": "kanamarlapudi", + "image_24": "https://avatars.slack-edge.com/2023-10-20/6067337928646_ac6b9c556b2a0741b284_24.jpg", + "image_32": "https://avatars.slack-edge.com/2023-10-20/6067337928646_ac6b9c556b2a0741b284_32.jpg", + "image_48": "https://avatars.slack-edge.com/2023-10-20/6067337928646_ac6b9c556b2a0741b284_48.jpg", + "image_72": "https://avatars.slack-edge.com/2023-10-20/6067337928646_ac6b9c556b2a0741b284_72.jpg", + "image_192": "https://avatars.slack-edge.com/2023-10-20/6067337928646_ac6b9c556b2a0741b284_192.jpg", + "image_512": "https://avatars.slack-edge.com/2023-10-20/6067337928646_ac6b9c556b2a0741b284_512.jpg", + "image_1024": "https://avatars.slack-edge.com/2023-10-20/6067337928646_ac6b9c556b2a0741b284_1024.jpg", + "status_text_canonical": "", + "team": "T01CWUYP5AR" + }, + "is_admin": false, + "is_owner": false, + "is_primary_owner": false, + "is_restricted": false, + "is_ultra_restricted": false, + "is_bot": false, + "is_app_user": false, + "updated": 1697840200, + "is_email_confirmed": true, + "has_2fa": false, + "who_can_share_contact_card": "EVERYONE" + }, + "U05KCF3EEUR": { + "id": "U05KCF3EEUR", + "team_id": "T01CWUYP5AR", + "name": "savansharan_navalgi", + "deleted": false, + "color": "a72f79", + "real_name": "savan", + "tz": "Asia/Kolkata", + "tz_label": "India Standard Time", + "tz_offset": 19800, + "profile": { + "title": "", + "phone": "", + "skype": "", + "real_name": "savan", + "real_name_normalized": "savan", + "display_name": "savan", + "display_name_normalized": "savan", + "fields": {}, + "status_text": "", + "status_emoji": "", + "status_emoji_display_info": [], + "status_expiration": 0, + "avatar_hash": "g9ec77e3c487", + "email": "SavanSharan_Navalgi@intuit.com", + "first_name": "savan", + "last_name": "", + "image_24": "https://secure.gravatar.com/avatar/9ec77e3c48747d0e4ba028cd952008b7.jpg?s=24&d=https%3A%2F%2Fa.slack-edge.com%2Fdf10d%2Fimg%2Favatars%2Fava_0009-24.png", + "image_32": "https://secure.gravatar.com/avatar/9ec77e3c48747d0e4ba028cd952008b7.jpg?s=32&d=https%3A%2F%2Fa.slack-edge.com%2Fdf10d%2Fimg%2Favatars%2Fava_0009-32.png", + "image_48": "https://secure.gravatar.com/avatar/9ec77e3c48747d0e4ba028cd952008b7.jpg?s=48&d=https%3A%2F%2Fa.slack-edge.com%2Fdf10d%2Fimg%2Favatars%2Fava_0009-48.png", + "image_72": "https://secure.gravatar.com/avatar/9ec77e3c48747d0e4ba028cd952008b7.jpg?s=72&d=https%3A%2F%2Fa.slack-edge.com%2Fdf10d%2Fimg%2Favatars%2Fava_0009-72.png", + "image_192": "https://secure.gravatar.com/avatar/9ec77e3c48747d0e4ba028cd952008b7.jpg?s=192&d=https%3A%2F%2Fa.slack-edge.com%2Fdf10d%2Fimg%2Favatars%2Fava_0009-192.png", + "image_512": "https://secure.gravatar.com/avatar/9ec77e3c48747d0e4ba028cd952008b7.jpg?s=512&d=https%3A%2F%2Fa.slack-edge.com%2Fdf10d%2Fimg%2Favatars%2Fava_0009-512.png", + "status_text_canonical": "", + "team": "T01CWUYP5AR" + }, + "is_admin": false, + "is_owner": false, + "is_primary_owner": false, + "is_restricted": false, + "is_ultra_restricted": false, + "is_bot": false, + "is_app_user": false, + "updated": 1690434999, + "is_email_confirmed": true, + "has_2fa": false, + "who_can_share_contact_card": "EVERYONE" + }, + "U03D8K119LJ": { + "id": "U03D8K119LJ", + "team_id": "T01CWUYP5AR", + "name": "matthewparas2020", + "deleted": false, + "color": "5b89d5", + "real_name": "Matthew Paras", + "tz": "America/Los_Angeles", + "tz_label": "Pacific Standard Time", + "tz_offset": -28800, + "profile": { + "title": "", + "phone": "", + "skype": "", + "real_name": "Matthew Paras", + "real_name_normalized": "Matthew Paras", + "display_name": "Matthew Paras", + "display_name_normalized": "Matthew Paras", + "fields": {}, + "status_text": "", + "status_emoji": "", + "status_emoji_display_info": [], + "status_expiration": 0, + "avatar_hash": "e96b6372b16b", + "image_original": "https://avatars.slack-edge.com/2022-04-27/3447754077317_e96b6372b16bba3317b8_original.png", + "is_custom_image": true, + "email": "matthewparas2020@u.northwestern.edu", + "first_name": "Matthew", + "last_name": "Paras", + "image_24": "https://avatars.slack-edge.com/2022-04-27/3447754077317_e96b6372b16bba3317b8_24.png", + "image_32": "https://avatars.slack-edge.com/2022-04-27/3447754077317_e96b6372b16bba3317b8_32.png", + "image_48": "https://avatars.slack-edge.com/2022-04-27/3447754077317_e96b6372b16bba3317b8_48.png", + "image_72": "https://avatars.slack-edge.com/2022-04-27/3447754077317_e96b6372b16bba3317b8_72.png", + "image_192": "https://avatars.slack-edge.com/2022-04-27/3447754077317_e96b6372b16bba3317b8_192.png", + "image_512": "https://avatars.slack-edge.com/2022-04-27/3447754077317_e96b6372b16bba3317b8_512.png", + "image_1024": "https://avatars.slack-edge.com/2022-04-27/3447754077317_e96b6372b16bba3317b8_1024.png", + "status_text_canonical": "", + "team": "T01CWUYP5AR" + }, + "is_admin": false, + "is_owner": false, + "is_primary_owner": false, + "is_restricted": false, + "is_ultra_restricted": false, + "is_bot": false, + "is_app_user": false, + "updated": 1651098588, + "is_email_confirmed": true, + "has_2fa": false, + "who_can_share_contact_card": "EVERYONE" + }, + "U0616K9TSTZ": { + "id": "U0616K9TSTZ", + "team_id": "T01CWUYP5AR", + "name": "ankit.goods10", + "deleted": false, + "color": "385a86", + "real_name": "ankit jain", + "tz": "Asia/Muscat", + "tz_label": "Gulf Standard Time", + "tz_offset": 14400, + "profile": { + "title": "", + "phone": "", + "skype": "", + "real_name": "ankit jain", + "real_name_normalized": "ankit jain", + "display_name": "ankit jain", + "display_name_normalized": "ankit jain", + "fields": {}, + "status_text": "", + "status_emoji": "", + "status_emoji_display_info": [], + "status_expiration": 0, + "avatar_hash": "76e78615ea7a", + "image_original": "https://avatars.slack-edge.com/2023-10-17/6057639084036_76e78615ea7aa1e80e98_original.jpg", + "is_custom_image": true, + "email": "ankit.goods10@gmail.com", + "first_name": "ankit", + "last_name": "jain", + "image_24": "https://avatars.slack-edge.com/2023-10-17/6057639084036_76e78615ea7aa1e80e98_24.jpg", + "image_32": "https://avatars.slack-edge.com/2023-10-17/6057639084036_76e78615ea7aa1e80e98_32.jpg", + "image_48": "https://avatars.slack-edge.com/2023-10-17/6057639084036_76e78615ea7aa1e80e98_48.jpg", + "image_72": "https://avatars.slack-edge.com/2023-10-17/6057639084036_76e78615ea7aa1e80e98_72.jpg", + "image_192": "https://avatars.slack-edge.com/2023-10-17/6057639084036_76e78615ea7aa1e80e98_192.jpg", + "image_512": "https://avatars.slack-edge.com/2023-10-17/6057639084036_76e78615ea7aa1e80e98_512.jpg", + "image_1024": "https://avatars.slack-edge.com/2023-10-17/6057639084036_76e78615ea7aa1e80e98_1024.jpg", + "status_text_canonical": "", + "team": "T01CWUYP5AR" + }, + "is_admin": false, + "is_owner": false, + "is_primary_owner": false, + "is_restricted": false, + "is_ultra_restricted": false, + "is_bot": false, + "is_app_user": false, + "updated": 1697597129, + "is_email_confirmed": true, + "has_2fa": false, + "who_can_share_contact_card": "EVERYONE" + }, + "U04EZ2LPDV4": { + "id": "U04EZ2LPDV4", + "team_id": "T01CWUYP5AR", + "name": "anirudh.shrinivason", + "deleted": false, + "color": "9f69e7", + "real_name": "Anirudh Shrinivason", + "tz": "Asia/Kuala_Lumpur", + "tz_label": "Singapore Standard Time", + "tz_offset": 28800, + "profile": { + "title": "", + "phone": "", + "skype": "", + "real_name": "Anirudh Shrinivason", + "real_name_normalized": "Anirudh Shrinivason", + "display_name": "Anirudh Shrinivason", + "display_name_normalized": "Anirudh Shrinivason", + "fields": {}, + "status_text": "", + "status_emoji": "", + "status_emoji_display_info": [], + "status_expiration": 1683043199, + "avatar_hash": "0d8f6e5a3065", + "image_original": "https://avatars.slack-edge.com/2023-05-22/5302055381746_0d8f6e5a3065f9038939_original.jpg", + "is_custom_image": true, + "email": "anirudh.shrinivason@grabtaxi.com", + "huddle_state": "default_unset", + "first_name": "Anirudh", + "last_name": "Shrinivason", + "image_24": "https://avatars.slack-edge.com/2023-05-22/5302055381746_0d8f6e5a3065f9038939_24.jpg", + "image_32": "https://avatars.slack-edge.com/2023-05-22/5302055381746_0d8f6e5a3065f9038939_32.jpg", + "image_48": "https://avatars.slack-edge.com/2023-05-22/5302055381746_0d8f6e5a3065f9038939_48.jpg", + "image_72": "https://avatars.slack-edge.com/2023-05-22/5302055381746_0d8f6e5a3065f9038939_72.jpg", + "image_192": "https://avatars.slack-edge.com/2023-05-22/5302055381746_0d8f6e5a3065f9038939_192.jpg", + "image_512": "https://avatars.slack-edge.com/2023-05-22/5302055381746_0d8f6e5a3065f9038939_512.jpg", + "image_1024": "https://avatars.slack-edge.com/2023-05-22/5302055381746_0d8f6e5a3065f9038939_1024.jpg", + "status_text_canonical": "", + "team": "T01CWUYP5AR" + }, + "is_admin": false, + "is_owner": false, + "is_primary_owner": false, + "is_restricted": false, + "is_ultra_restricted": false, + "is_bot": false, + "is_app_user": false, + "updated": 1684763470, + "is_email_confirmed": true, + "has_2fa": false, + "who_can_share_contact_card": "EVERYONE" + }, + "U05HK41VCH1": { + "id": "U05HK41VCH1", + "team_id": "T01CWUYP5AR", + "name": "madhav.kakumani", + "deleted": false, + "color": "e06b56", + "real_name": "Madhav Kakumani", + "tz": "Europe/London", + "tz_label": "Greenwich Mean Time", + "tz_offset": 0, + "profile": { + "title": "", + "phone": "", + "skype": "", + "real_name": "Madhav Kakumani", + "real_name_normalized": "Madhav Kakumani", + "display_name": "Madhav Kakumani", + "display_name_normalized": "Madhav Kakumani", + "fields": {}, + "status_text": "", + "status_emoji": "", + "status_emoji_display_info": [], + "status_expiration": 0, + "avatar_hash": "558c36409436", + "image_original": "https://avatars.slack-edge.com/2023-07-18/5597252400930_558c3640943668a0198e_original.png", + "is_custom_image": true, + "email": "madhav.kakumani@6point6.co.uk", + "first_name": "Madhav", + "last_name": "Kakumani", + "image_24": "https://avatars.slack-edge.com/2023-07-18/5597252400930_558c3640943668a0198e_24.png", + "image_32": "https://avatars.slack-edge.com/2023-07-18/5597252400930_558c3640943668a0198e_32.png", + "image_48": "https://avatars.slack-edge.com/2023-07-18/5597252400930_558c3640943668a0198e_48.png", + "image_72": "https://avatars.slack-edge.com/2023-07-18/5597252400930_558c3640943668a0198e_72.png", + "image_192": "https://avatars.slack-edge.com/2023-07-18/5597252400930_558c3640943668a0198e_192.png", + "image_512": "https://avatars.slack-edge.com/2023-07-18/5597252400930_558c3640943668a0198e_512.png", + "image_1024": "https://avatars.slack-edge.com/2023-07-18/5597252400930_558c3640943668a0198e_1024.png", + "status_text_canonical": "", + "team": "T01CWUYP5AR" + }, + "is_admin": false, + "is_owner": false, + "is_primary_owner": false, + "is_restricted": false, + "is_ultra_restricted": false, + "is_bot": false, + "is_app_user": false, + "updated": 1689687879, + "is_email_confirmed": true, + "has_2fa": false, + "who_can_share_contact_card": "EVERYONE" + }, + "U05QL7LN2GH": { + "id": "U05QL7LN2GH", + "team_id": "T01CWUYP5AR", + "name": "jeevan", + "deleted": false, + "color": "8d4b84", + "real_name": "Guntaka Jeevan Paul", + "tz": "Asia/Kolkata", + "tz_label": "India Standard Time", + "tz_offset": 19800, + "profile": { + "title": "", + "phone": "", + "skype": "", + "real_name": "Guntaka Jeevan Paul", + "real_name_normalized": "Guntaka Jeevan Paul", + "display_name": "Guntaka Jeevan Paul", + "display_name_normalized": "Guntaka Jeevan Paul", + "fields": {}, + "status_text": "", + "status_emoji": "", + "status_emoji_display_info": [], + "status_expiration": 0, + "avatar_hash": "5ef61b937b8b", + "image_original": "https://avatars.slack-edge.com/2023-08-30/5820708969061_5ef61b937b8b8e9b2d6e_original.png", + "is_custom_image": true, + "email": "jeevan@acceldata.io", + "first_name": "Guntaka", + "last_name": "Jeevan Paul", + "image_24": "https://avatars.slack-edge.com/2023-08-30/5820708969061_5ef61b937b8b8e9b2d6e_24.png", + "image_32": "https://avatars.slack-edge.com/2023-08-30/5820708969061_5ef61b937b8b8e9b2d6e_32.png", + "image_48": "https://avatars.slack-edge.com/2023-08-30/5820708969061_5ef61b937b8b8e9b2d6e_48.png", + "image_72": "https://avatars.slack-edge.com/2023-08-30/5820708969061_5ef61b937b8b8e9b2d6e_72.png", + "image_192": "https://avatars.slack-edge.com/2023-08-30/5820708969061_5ef61b937b8b8e9b2d6e_192.png", + "image_512": "https://avatars.slack-edge.com/2023-08-30/5820708969061_5ef61b937b8b8e9b2d6e_512.png", + "image_1024": "https://avatars.slack-edge.com/2023-08-30/5820708969061_5ef61b937b8b8e9b2d6e_1024.png", + "status_text_canonical": "", + "team": "T01CWUYP5AR" + }, + "is_admin": false, + "is_owner": false, + "is_primary_owner": false, + "is_restricted": false, + "is_ultra_restricted": false, + "is_bot": false, + "is_app_user": false, + "updated": 1693392815, + "is_email_confirmed": true, + "has_2fa": false, + "who_can_share_contact_card": "EVERYONE" + }, + "U021QJMRP47": { + "id": "U021QJMRP47", + "team_id": "T01CWUYP5AR", + "name": "drew215", + "deleted": false, + "color": "e0a729", + "real_name": "Drew Bittenbender", + "tz": "America/New_York", + "tz_label": "Eastern Standard Time", + "tz_offset": -18000, + "profile": { + "title": "", + "phone": "", + "skype": "", + "real_name": "Drew Bittenbender", + "real_name_normalized": "Drew Bittenbender", + "display_name": "Drew Bittenbender", + "display_name_normalized": "Drew Bittenbender", + "fields": {}, + "status_text": "", + "status_emoji": "", + "status_emoji_display_info": [], + "status_expiration": 0, + "avatar_hash": "g557c9e0a7ba", + "email": "drew@salt.io", + "image_24": "https://secure.gravatar.com/avatar/557c9e0a7ba1b608e34ffe64ce219a49.jpg?s=24&d=https%3A%2F%2Fa.slack-edge.com%2Fdf10d%2Fimg%2Favatars%2Fava_0017-24.png", + "image_32": "https://secure.gravatar.com/avatar/557c9e0a7ba1b608e34ffe64ce219a49.jpg?s=32&d=https%3A%2F%2Fa.slack-edge.com%2Fdf10d%2Fimg%2Favatars%2Fava_0017-32.png", + "image_48": "https://secure.gravatar.com/avatar/557c9e0a7ba1b608e34ffe64ce219a49.jpg?s=48&d=https%3A%2F%2Fa.slack-edge.com%2Fdf10d%2Fimg%2Favatars%2Fava_0017-48.png", + "image_72": "https://secure.gravatar.com/avatar/557c9e0a7ba1b608e34ffe64ce219a49.jpg?s=72&d=https%3A%2F%2Fa.slack-edge.com%2Fdf10d%2Fimg%2Favatars%2Fava_0017-72.png", + "image_192": "https://secure.gravatar.com/avatar/557c9e0a7ba1b608e34ffe64ce219a49.jpg?s=192&d=https%3A%2F%2Fa.slack-edge.com%2Fdf10d%2Fimg%2Favatars%2Fava_0017-192.png", + "image_512": "https://secure.gravatar.com/avatar/557c9e0a7ba1b608e34ffe64ce219a49.jpg?s=512&d=https%3A%2F%2Fa.slack-edge.com%2Fdf10d%2Fimg%2Favatars%2Fava_0017-512.png", + "status_text_canonical": "", + "team": "T01CWUYP5AR" + }, + "is_admin": false, + "is_owner": false, + "is_primary_owner": false, + "is_restricted": false, + "is_ultra_restricted": false, + "is_bot": false, + "is_app_user": false, + "updated": 1660506044, + "is_email_confirmed": true, + "has_2fa": false, + "who_can_share_contact_card": "EVERYONE" + }, + "U01HVNU6A4C": { + "id": "U01HVNU6A4C", + "team_id": "T01CWUYP5AR", + "name": "mars", + "deleted": false, + "color": "9e3997", + "real_name": "Mars Lan", + "tz": "America/Los_Angeles", + "tz_label": "Pacific Standard Time", + "tz_offset": -28800, + "profile": { + "title": "", + "phone": "", + "skype": "", + "real_name": "Mars Lan", + "real_name_normalized": "Mars Lan", + "display_name": "Mars Lan", + "display_name_normalized": "Mars Lan", + "fields": {}, + "status_text": "", + "status_emoji": "", + "status_emoji_display_info": [], + "status_expiration": 0, + "avatar_hash": "e70f0b60ad8c", + "image_original": "https://avatars.slack-edge.com/2020-12-28/1598525158821_e70f0b60ad8c72e52398_original.jpg", + "is_custom_image": true, + "email": "mars@metaphor.io", + "image_24": "https://avatars.slack-edge.com/2020-12-28/1598525158821_e70f0b60ad8c72e52398_24.jpg", + "image_32": "https://avatars.slack-edge.com/2020-12-28/1598525158821_e70f0b60ad8c72e52398_32.jpg", + "image_48": "https://avatars.slack-edge.com/2020-12-28/1598525158821_e70f0b60ad8c72e52398_48.jpg", + "image_72": "https://avatars.slack-edge.com/2020-12-28/1598525158821_e70f0b60ad8c72e52398_72.jpg", + "image_192": "https://avatars.slack-edge.com/2020-12-28/1598525158821_e70f0b60ad8c72e52398_192.jpg", + "image_512": "https://avatars.slack-edge.com/2020-12-28/1598525158821_e70f0b60ad8c72e52398_512.jpg", + "image_1024": "https://avatars.slack-edge.com/2020-12-28/1598525158821_e70f0b60ad8c72e52398_1024.jpg", + "status_text_canonical": "", + "team": "T01CWUYP5AR" + }, + "is_admin": false, + "is_owner": false, + "is_primary_owner": false, + "is_restricted": false, + "is_ultra_restricted": false, + "is_bot": false, + "is_app_user": false, + "updated": 1609167164, + "is_email_confirmed": true, + "has_2fa": false, + "who_can_share_contact_card": "EVERYONE" + }, + "U01DCLP0GU9": { + "id": "U01DCLP0GU9", + "team_id": "T01CWUYP5AR", + "name": "julien", + "deleted": false, + "color": "9f69e7", + "real_name": "Julien Le Dem", + "tz": "America/Los_Angeles", + "tz_label": "Pacific Standard Time", + "tz_offset": -28800, + "profile": { + "title": "Astronomer Chief Architect and Datakin co-founder, OpenLineage project lead", + "phone": "", + "skype": "", + "real_name": "Julien Le Dem", + "real_name_normalized": "Julien Le Dem", + "display_name": "", + "display_name_normalized": "", + "fields": { + "Xf03UL3S8CP8": { + "value": "Astronomer Chief Architect and Datakin co-founder, OpenLineage project lead", + "alt": "" + } + }, + "status_text": "Vacationing", + "status_emoji": ":palm_tree:", + "status_emoji_display_info": [ + { + "emoji_name": "palm_tree", + "display_url": "https://a.slack-edge.com/production-standard-emoji-assets/14.0/apple-large/1f334.png", + "unicode": "1f334" + } + ], + "status_expiration": 0, + "avatar_hash": "60dbc1781564", + "image_original": "https://avatars.slack-edge.com/2020-11-03/1467961410374_60dbc1781564d114f9e1_original.jpg", + "is_custom_image": true, + "email": "julien@apache.org", + "first_name": "Julien", + "last_name": "Le Dem", + "image_24": "https://avatars.slack-edge.com/2020-11-03/1467961410374_60dbc1781564d114f9e1_24.jpg", + "image_32": "https://avatars.slack-edge.com/2020-11-03/1467961410374_60dbc1781564d114f9e1_32.jpg", + "image_48": "https://avatars.slack-edge.com/2020-11-03/1467961410374_60dbc1781564d114f9e1_48.jpg", + "image_72": "https://avatars.slack-edge.com/2020-11-03/1467961410374_60dbc1781564d114f9e1_72.jpg", + "image_192": "https://avatars.slack-edge.com/2020-11-03/1467961410374_60dbc1781564d114f9e1_192.jpg", + "image_512": "https://avatars.slack-edge.com/2020-11-03/1467961410374_60dbc1781564d114f9e1_512.jpg", + "image_1024": "https://avatars.slack-edge.com/2020-11-03/1467961410374_60dbc1781564d114f9e1_1024.jpg", + "status_text_canonical": "Vacationing", + "team": "T01CWUYP5AR" + }, + "is_admin": true, + "is_owner": true, + "is_primary_owner": true, + "is_restricted": false, + "is_ultra_restricted": false, + "is_bot": false, + "is_app_user": false, + "updated": 1700183282, + "is_email_confirmed": true, + "has_2fa": false, + "who_can_share_contact_card": "EVERYONE" + }, + "U05FLJE4GDU": { + "id": "U05FLJE4GDU", + "team_id": "T01CWUYP5AR", + "name": "damien.hawes", + "deleted": false, + "color": "e96699", + "real_name": "Damien Hawes", + "tz": "Europe/Amsterdam", + "tz_label": "Central European Time", + "tz_offset": 3600, + "profile": { + "title": "", + "phone": "", + "skype": "", + "real_name": "Damien Hawes", + "real_name_normalized": "Damien Hawes", + "display_name": "Damien Hawes", + "display_name_normalized": "Damien Hawes", + "fields": { + "Xf01NDQXPW6A": { + "value": "https://github.com/d-m-h", + "alt": "GitHub Profile" + } + }, + "status_text": "", + "status_emoji": "", + "status_emoji_display_info": [], + "status_expiration": 0, + "avatar_hash": "6f58b97494c9", + "image_original": "https://avatars.slack-edge.com/2023-07-06/5530626222838_6f58b97494c98ee81c7d_original.png", + "is_custom_image": true, + "email": "damien.hawes@booking.com", + "first_name": "Damien", + "last_name": "Hawes", + "image_24": "https://avatars.slack-edge.com/2023-07-06/5530626222838_6f58b97494c98ee81c7d_24.png", + "image_32": "https://avatars.slack-edge.com/2023-07-06/5530626222838_6f58b97494c98ee81c7d_32.png", + "image_48": "https://avatars.slack-edge.com/2023-07-06/5530626222838_6f58b97494c98ee81c7d_48.png", + "image_72": "https://avatars.slack-edge.com/2023-07-06/5530626222838_6f58b97494c98ee81c7d_72.png", + "image_192": "https://avatars.slack-edge.com/2023-07-06/5530626222838_6f58b97494c98ee81c7d_192.png", + "image_512": "https://avatars.slack-edge.com/2023-07-06/5530626222838_6f58b97494c98ee81c7d_512.png", + "image_1024": "https://avatars.slack-edge.com/2023-07-06/5530626222838_6f58b97494c98ee81c7d_1024.png", + "status_text_canonical": "", + "team": "T01CWUYP5AR" + }, + "is_admin": false, + "is_owner": false, + "is_primary_owner": false, + "is_restricted": false, + "is_ultra_restricted": false, + "is_bot": false, + "is_app_user": false, + "updated": 1688644886, + "is_email_confirmed": true, + "has_2fa": false, + "who_can_share_contact_card": "EVERYONE" + }, + "U05TZE47F2S": { + "id": "U05TZE47F2S", + "team_id": "T01CWUYP5AR", + "name": "slack1950", + "deleted": false, + "color": "2b6836", + "real_name": "Erik Alfthan", + "tz": "Europe/Amsterdam", + "tz_label": "Central European Time", + "tz_offset": 3600, + "profile": { + "title": "", + "phone": "", + "skype": "", + "real_name": "Erik Alfthan", + "real_name_normalized": "Erik Alfthan", + "display_name": "Erik Alfthan", + "display_name_normalized": "Erik Alfthan", + "fields": {}, + "status_text": "", + "status_emoji": "", + "status_emoji_display_info": [], + "status_expiration": 0, + "avatar_hash": "g17fe479d7c0", + "email": "slack@alfthan.eu", + "first_name": "Erik", + "last_name": "Alfthan", + "image_24": "https://secure.gravatar.com/avatar/17fe479d7c0a30a4a4e927793af59c14.jpg?s=24&d=https%3A%2F%2Fa.slack-edge.com%2Fdf10d%2Fimg%2Favatars%2Fava_0023-24.png", + "image_32": "https://secure.gravatar.com/avatar/17fe479d7c0a30a4a4e927793af59c14.jpg?s=32&d=https%3A%2F%2Fa.slack-edge.com%2Fdf10d%2Fimg%2Favatars%2Fava_0023-32.png", + "image_48": "https://secure.gravatar.com/avatar/17fe479d7c0a30a4a4e927793af59c14.jpg?s=48&d=https%3A%2F%2Fa.slack-edge.com%2Fdf10d%2Fimg%2Favatars%2Fava_0023-48.png", + "image_72": "https://secure.gravatar.com/avatar/17fe479d7c0a30a4a4e927793af59c14.jpg?s=72&d=https%3A%2F%2Fa.slack-edge.com%2Fdf10d%2Fimg%2Favatars%2Fava_0023-72.png", + "image_192": "https://secure.gravatar.com/avatar/17fe479d7c0a30a4a4e927793af59c14.jpg?s=192&d=https%3A%2F%2Fa.slack-edge.com%2Fdf10d%2Fimg%2Favatars%2Fava_0023-192.png", + "image_512": "https://secure.gravatar.com/avatar/17fe479d7c0a30a4a4e927793af59c14.jpg?s=512&d=https%3A%2F%2Fa.slack-edge.com%2Fdf10d%2Fimg%2Favatars%2Fava_0023-512.png", + "status_text_canonical": "", + "team": "T01CWUYP5AR" + }, + "is_admin": false, + "is_owner": false, + "is_primary_owner": false, + "is_restricted": false, + "is_ultra_restricted": false, + "is_bot": false, + "is_app_user": false, + "updated": 1695828947, + "is_email_confirmed": true, + "has_2fa": false, + "who_can_share_contact_card": "EVERYONE" + }, + "U05A1D80QKF": { + "id": "U05A1D80QKF", + "team_id": "T01CWUYP5AR", + "name": "suraj.gupta", + "deleted": false, + "color": "a63024", + "real_name": "Suraj Gupta", + "tz": "Asia/Kolkata", + "tz_label": "India Standard Time", + "tz_offset": 19800, + "profile": { + "title": "", + "phone": "", + "skype": "", + "real_name": "Suraj Gupta", + "real_name_normalized": "Suraj Gupta", + "display_name": "Suraj Gupta", + "display_name_normalized": "Suraj Gupta", + "fields": {}, + "status_text": "", + "status_emoji": "", + "status_emoji_display_info": [], + "status_expiration": 0, + "avatar_hash": "b3c546a0a2ea", + "image_original": "https://avatars.slack-edge.com/2023-09-25/5946511342292_b3c546a0a2eac62dae08_original.jpg", + "is_custom_image": true, + "email": "suraj.gupta@atlan.com", + "first_name": "Suraj", + "last_name": "Gupta", + "image_24": "https://avatars.slack-edge.com/2023-09-25/5946511342292_b3c546a0a2eac62dae08_24.jpg", + "image_32": "https://avatars.slack-edge.com/2023-09-25/5946511342292_b3c546a0a2eac62dae08_32.jpg", + "image_48": "https://avatars.slack-edge.com/2023-09-25/5946511342292_b3c546a0a2eac62dae08_48.jpg", + "image_72": "https://avatars.slack-edge.com/2023-09-25/5946511342292_b3c546a0a2eac62dae08_72.jpg", + "image_192": "https://avatars.slack-edge.com/2023-09-25/5946511342292_b3c546a0a2eac62dae08_192.jpg", + "image_512": "https://avatars.slack-edge.com/2023-09-25/5946511342292_b3c546a0a2eac62dae08_512.jpg", + "image_1024": "https://avatars.slack-edge.com/2023-09-25/5946511342292_b3c546a0a2eac62dae08_1024.jpg", + "status_text_canonical": "", + "team": "T01CWUYP5AR" + }, + "is_admin": false, + "is_owner": false, + "is_primary_owner": false, + "is_restricted": false, + "is_ultra_restricted": false, + "is_bot": false, + "is_app_user": false, + "updated": 1695640006, + "is_email_confirmed": true, + "has_2fa": false, + "who_can_share_contact_card": "EVERYONE" + }, + "U05SMTVPPL3": { + "id": "U05SMTVPPL3", + "team_id": "T01CWUYP5AR", + "name": "sangeeta", + "deleted": false, + "color": "dc7dbb", + "real_name": "Sangeeta Mishra", + "tz": "Asia/Kolkata", + "tz_label": "India Standard Time", + "tz_offset": 19800, + "profile": { + "title": "", + "phone": "", + "skype": "", + "real_name": "Sangeeta Mishra", + "real_name_normalized": "Sangeeta Mishra", + "display_name": "Sangeeta Mishra", + "display_name_normalized": "Sangeeta Mishra", + "fields": {}, + "status_text": "", + "status_emoji": "", + "status_emoji_display_info": [], + "status_expiration": 0, + "avatar_hash": "00fc6497e7eb", + "image_original": "https://avatars.slack-edge.com/2023-09-16/5918683629649_00fc6497e7ebbd764ef4_original.png", + "is_custom_image": true, + "email": "sangeeta@acceldata.io", + "first_name": "Sangeeta", + "last_name": "Mishra", + "image_24": "https://avatars.slack-edge.com/2023-09-16/5918683629649_00fc6497e7ebbd764ef4_24.png", + "image_32": "https://avatars.slack-edge.com/2023-09-16/5918683629649_00fc6497e7ebbd764ef4_32.png", + "image_48": "https://avatars.slack-edge.com/2023-09-16/5918683629649_00fc6497e7ebbd764ef4_48.png", + "image_72": "https://avatars.slack-edge.com/2023-09-16/5918683629649_00fc6497e7ebbd764ef4_72.png", + "image_192": "https://avatars.slack-edge.com/2023-09-16/5918683629649_00fc6497e7ebbd764ef4_192.png", + "image_512": "https://avatars.slack-edge.com/2023-09-16/5918683629649_00fc6497e7ebbd764ef4_512.png", + "image_1024": "https://avatars.slack-edge.com/2023-09-16/5918683629649_00fc6497e7ebbd764ef4_1024.png", + "status_text_canonical": "", + "team": "T01CWUYP5AR" + }, + "is_admin": false, + "is_owner": false, + "is_primary_owner": false, + "is_restricted": false, + "is_ultra_restricted": false, + "is_bot": false, + "is_app_user": false, + "updated": 1694852931, + "is_email_confirmed": true, + "has_2fa": false, + "who_can_share_contact_card": "EVERYONE" + }, + "U05HFGKEYVB": { + "id": "U05HFGKEYVB", + "team_id": "T01CWUYP5AR", + "name": "juan_luis_cano", + "deleted": false, + "color": "43761b", + "real_name": "Juan Luis Cano Rodríguez", + "tz": "Europe/Amsterdam", + "tz_label": "Central European Time", + "tz_offset": 3600, + "profile": { + "title": "Product Manager for Kedro at QuantumBlack, AI by McKinsey", + "phone": "", + "skype": "", + "real_name": "Juan Luis Cano Rodríguez", + "real_name_normalized": "Juan Luis Cano Rodriguez", + "display_name": "Juan Luis Cano Rodríguez", + "display_name_normalized": "Juan Luis Cano Rodriguez", + "fields": { + "Xf03UL3S8CP8": { + "value": "Product Manager for Kedro at QuantumBlack, AI by McKinsey", + "alt": "" + }, + "Xf01NDQXPW6A": { + "value": "https://github.com/astrojuanlu", + "alt": "@astrojuanlu" + }, + "Xf03JNKWMZ6J": { + "value": "/xuan luis/", + "alt": "" + } + }, + "status_text": "", + "status_emoji": "", + "status_emoji_display_info": [], + "status_expiration": 0, + "avatar_hash": "7827e7e405e2", + "image_original": "https://avatars.slack-edge.com/2023-08-02/5678140113618_7827e7e405e2f18605c2_original.jpg", + "is_custom_image": true, + "email": "juan_luis_cano@mckinsey.com", + "first_name": "Juan", + "last_name": "Luis Cano Rodríguez", + "image_24": "https://avatars.slack-edge.com/2023-08-02/5678140113618_7827e7e405e2f18605c2_24.jpg", + "image_32": "https://avatars.slack-edge.com/2023-08-02/5678140113618_7827e7e405e2f18605c2_32.jpg", + "image_48": "https://avatars.slack-edge.com/2023-08-02/5678140113618_7827e7e405e2f18605c2_48.jpg", + "image_72": "https://avatars.slack-edge.com/2023-08-02/5678140113618_7827e7e405e2f18605c2_72.jpg", + "image_192": "https://avatars.slack-edge.com/2023-08-02/5678140113618_7827e7e405e2f18605c2_192.jpg", + "image_512": "https://avatars.slack-edge.com/2023-08-02/5678140113618_7827e7e405e2f18605c2_512.jpg", + "image_1024": "https://avatars.slack-edge.com/2023-08-02/5678140113618_7827e7e405e2f18605c2_1024.jpg", + "status_text_canonical": "", + "team": "T01CWUYP5AR" + }, + "is_admin": false, + "is_owner": false, + "is_primary_owner": false, + "is_restricted": false, + "is_ultra_restricted": false, + "is_bot": false, + "is_app_user": false, + "updated": 1699314500, + "is_email_confirmed": true, + "has_2fa": false, + "who_can_share_contact_card": "EVERYONE" + }, + "U05SQGH8DV4": { + "id": "U05SQGH8DV4", + "team_id": "T01CWUYP5AR", + "name": "sarathch", + "deleted": false, + "color": "73769d", + "real_name": "sarathch", + "tz": "Asia/Kolkata", + "tz_label": "India Standard Time", + "tz_offset": 19800, + "profile": { + "title": "", + "phone": "", + "skype": "", + "real_name": "sarathch", + "real_name_normalized": "sarathch", + "display_name": "sarathch", + "display_name_normalized": "sarathch", + "fields": {}, + "status_text": "", + "status_emoji": "", + "status_emoji_display_info": [], + "status_expiration": 0, + "avatar_hash": "g4916b069d83", + "email": "sarathch@hpe.com", + "first_name": "sarathch", + "last_name": "", + "image_24": "https://secure.gravatar.com/avatar/4916b069d8350cf64eba820d01b8fd0c.jpg?s=24&d=https%3A%2F%2Fa.slack-edge.com%2Fdf10d%2Fimg%2Favatars%2Fava_0006-24.png", + "image_32": "https://secure.gravatar.com/avatar/4916b069d8350cf64eba820d01b8fd0c.jpg?s=32&d=https%3A%2F%2Fa.slack-edge.com%2Fdf10d%2Fimg%2Favatars%2Fava_0006-32.png", + "image_48": "https://secure.gravatar.com/avatar/4916b069d8350cf64eba820d01b8fd0c.jpg?s=48&d=https%3A%2F%2Fa.slack-edge.com%2Fdf10d%2Fimg%2Favatars%2Fava_0006-48.png", + "image_72": "https://secure.gravatar.com/avatar/4916b069d8350cf64eba820d01b8fd0c.jpg?s=72&d=https%3A%2F%2Fa.slack-edge.com%2Fdf10d%2Fimg%2Favatars%2Fava_0006-72.png", + "image_192": "https://secure.gravatar.com/avatar/4916b069d8350cf64eba820d01b8fd0c.jpg?s=192&d=https%3A%2F%2Fa.slack-edge.com%2Fdf10d%2Fimg%2Favatars%2Fava_0006-192.png", + "image_512": "https://secure.gravatar.com/avatar/4916b069d8350cf64eba820d01b8fd0c.jpg?s=512&d=https%3A%2F%2Fa.slack-edge.com%2Fdf10d%2Fimg%2Favatars%2Fava_0006-512.png", + "status_text_canonical": "", + "team": "T01CWUYP5AR" + }, + "is_admin": false, + "is_owner": false, + "is_primary_owner": false, + "is_restricted": false, + "is_ultra_restricted": false, + "is_bot": false, + "is_app_user": false, + "updated": 1695105737, + "is_email_confirmed": true, + "has_2fa": false, + "who_can_share_contact_card": "EVERYONE" + }, + "U05K8F1T887": { + "id": "U05K8F1T887", + "team_id": "T01CWUYP5AR", + "name": "terese", + "deleted": false, + "color": "bd9336", + "real_name": "Terese Larsson", + "tz": "Europe/Amsterdam", + "tz_label": "Central European Time", + "tz_offset": 3600, + "profile": { + "title": "", + "phone": "", + "skype": "", + "real_name": "Terese Larsson", + "real_name_normalized": "Terese Larsson", + "display_name": "Terese Larsson", + "display_name_normalized": "Terese Larsson", + "fields": {}, + "status_text": "", + "status_emoji": "", + "status_emoji_display_info": [], + "status_expiration": 0, + "avatar_hash": "7989ca00a3d3", + "image_original": "https://avatars.slack-edge.com/2023-07-31/5692752759088_7989ca00a3d39c7676dd_original.png", + "is_custom_image": true, + "email": "terese@jclab.se", + "first_name": "Terese", + "last_name": "Larsson", + "image_24": "https://avatars.slack-edge.com/2023-07-31/5692752759088_7989ca00a3d39c7676dd_24.png", + "image_32": "https://avatars.slack-edge.com/2023-07-31/5692752759088_7989ca00a3d39c7676dd_32.png", + "image_48": "https://avatars.slack-edge.com/2023-07-31/5692752759088_7989ca00a3d39c7676dd_48.png", + "image_72": "https://avatars.slack-edge.com/2023-07-31/5692752759088_7989ca00a3d39c7676dd_72.png", + "image_192": "https://avatars.slack-edge.com/2023-07-31/5692752759088_7989ca00a3d39c7676dd_192.png", + "image_512": "https://avatars.slack-edge.com/2023-07-31/5692752759088_7989ca00a3d39c7676dd_512.png", + "image_1024": "https://avatars.slack-edge.com/2023-07-31/5692752759088_7989ca00a3d39c7676dd_1024.png", + "status_text_canonical": "", + "team": "T01CWUYP5AR" + }, + "is_admin": false, + "is_owner": false, + "is_primary_owner": false, + "is_restricted": false, + "is_ultra_restricted": false, + "is_bot": false, + "is_app_user": false, + "updated": 1690872647, + "is_email_confirmed": true, + "has_2fa": false, + "who_can_share_contact_card": "EVERYONE" + }, + "U055N2GRT4P": { + "id": "U055N2GRT4P", + "team_id": "T01CWUYP5AR", + "name": "tatiana.alchueyr", + "deleted": false, + "color": "e7392d", + "real_name": "tati", + "tz": "Europe/London", + "tz_label": "Greenwich Mean Time", + "tz_offset": 0, + "profile": { + "title": "", + "phone": "", + "skype": "", + "real_name": "tati", + "real_name_normalized": "tati", + "display_name": "tati", + "display_name_normalized": "tati", + "fields": {}, + "status_text": "", + "status_emoji": "", + "status_emoji_display_info": [], + "status_expiration": 0, + "avatar_hash": "g21115d62a29", + "email": "tatiana.alchueyr@astronomer.io", + "first_name": "tati", + "last_name": "", + "image_24": "https://secure.gravatar.com/avatar/21115d62a29c683f1390c9dfca2367fe.jpg?s=24&d=https%3A%2F%2Fa.slack-edge.com%2Fdf10d%2Fimg%2Favatars%2Fava_0010-24.png", + "image_32": "https://secure.gravatar.com/avatar/21115d62a29c683f1390c9dfca2367fe.jpg?s=32&d=https%3A%2F%2Fa.slack-edge.com%2Fdf10d%2Fimg%2Favatars%2Fava_0010-32.png", + "image_48": "https://secure.gravatar.com/avatar/21115d62a29c683f1390c9dfca2367fe.jpg?s=48&d=https%3A%2F%2Fa.slack-edge.com%2Fdf10d%2Fimg%2Favatars%2Fava_0010-48.png", + "image_72": "https://secure.gravatar.com/avatar/21115d62a29c683f1390c9dfca2367fe.jpg?s=72&d=https%3A%2F%2Fa.slack-edge.com%2Fdf10d%2Fimg%2Favatars%2Fava_0010-72.png", + "image_192": "https://secure.gravatar.com/avatar/21115d62a29c683f1390c9dfca2367fe.jpg?s=192&d=https%3A%2F%2Fa.slack-edge.com%2Fdf10d%2Fimg%2Favatars%2Fava_0010-192.png", + "image_512": "https://secure.gravatar.com/avatar/21115d62a29c683f1390c9dfca2367fe.jpg?s=512&d=https%3A%2F%2Fa.slack-edge.com%2Fdf10d%2Fimg%2Favatars%2Fava_0010-512.png", + "status_text_canonical": "", + "team": "T01CWUYP5AR" + }, + "is_admin": false, + "is_owner": false, + "is_primary_owner": false, + "is_restricted": false, + "is_ultra_restricted": false, + "is_bot": false, + "is_app_user": false, + "updated": 1695383702, + "is_email_confirmed": true, + "has_2fa": false, + "who_can_share_contact_card": "EVERYONE" + }, + "U05QNRSQW1E": { + "id": "U05QNRSQW1E", + "team_id": "T01CWUYP5AR", + "name": "sarwatfatimam", + "deleted": false, + "color": "a2a5dc", + "real_name": "Sarwat Fatima", + "tz": "Asia/Karachi", + "tz_label": "Pakistan Standard Time", + "tz_offset": 18000, + "profile": { + "title": "", + "phone": "", + "skype": "", + "real_name": "Sarwat Fatima", + "real_name_normalized": "Sarwat Fatima", + "display_name": "Sarwat Fatima", + "display_name_normalized": "Sarwat Fatima", + "fields": {}, + "status_text": "", + "status_emoji": "", + "status_emoji_display_info": [], + "status_expiration": 0, + "avatar_hash": "30158cc69ec1", + "image_original": "https://avatars.slack-edge.com/2023-08-29/5817622531764_30158cc69ec15ed4a1c8_original.png", + "is_custom_image": true, + "email": "sarwatfatimam@gmail.com", + "huddle_state": "default_unset", + "huddle_state_expiration_ts": 0, + "first_name": "Sarwat", + "last_name": "Fatima", + "image_24": "https://avatars.slack-edge.com/2023-08-29/5817622531764_30158cc69ec15ed4a1c8_24.png", + "image_32": "https://avatars.slack-edge.com/2023-08-29/5817622531764_30158cc69ec15ed4a1c8_32.png", + "image_48": "https://avatars.slack-edge.com/2023-08-29/5817622531764_30158cc69ec15ed4a1c8_48.png", + "image_72": "https://avatars.slack-edge.com/2023-08-29/5817622531764_30158cc69ec15ed4a1c8_72.png", + "image_192": "https://avatars.slack-edge.com/2023-08-29/5817622531764_30158cc69ec15ed4a1c8_192.png", + "image_512": "https://avatars.slack-edge.com/2023-08-29/5817622531764_30158cc69ec15ed4a1c8_512.png", + "image_1024": "https://avatars.slack-edge.com/2023-08-29/5817622531764_30158cc69ec15ed4a1c8_1024.png", + "status_text_canonical": "", + "team": "T01CWUYP5AR" + }, + "is_admin": false, + "is_owner": false, + "is_primary_owner": false, + "is_restricted": false, + "is_ultra_restricted": false, + "is_bot": false, + "is_app_user": false, + "updated": 1693935133, + "is_email_confirmed": true, + "has_2fa": false, + "who_can_share_contact_card": "EVERYONE" + }, + "U0595Q78HUG": { + "id": "U0595Q78HUG", + "team_id": "T01CWUYP5AR", + "name": "gaborjbernat", + "deleted": false, + "color": "5a4592", + "real_name": "Bernat Gabor", + "tz": "America/Los_Angeles", + "tz_label": "Pacific Standard Time", + "tz_offset": -28800, + "profile": { + "title": "", + "phone": "", + "skype": "", + "real_name": "Bernat Gabor", + "real_name_normalized": "Bernat Gabor", + "display_name": "Bernat Gabor", + "display_name_normalized": "Bernat Gabor", + "fields": {}, + "status_text": "", + "status_emoji": "", + "status_emoji_display_info": [], + "status_expiration": 0, + "avatar_hash": "gf8cbaf456d5", + "email": "gaborjbernat@gmail.com", + "first_name": "Bernat", + "last_name": "Gabor", + "image_24": "https://secure.gravatar.com/avatar/f8cbaf456d5bed4fe78d758a5e8d27fc.jpg?s=24&d=https%3A%2F%2Fa.slack-edge.com%2Fdf10d%2Fimg%2Favatars%2Fava_0015-24.png", + "image_32": "https://secure.gravatar.com/avatar/f8cbaf456d5bed4fe78d758a5e8d27fc.jpg?s=32&d=https%3A%2F%2Fa.slack-edge.com%2Fdf10d%2Fimg%2Favatars%2Fava_0015-32.png", + "image_48": "https://secure.gravatar.com/avatar/f8cbaf456d5bed4fe78d758a5e8d27fc.jpg?s=48&d=https%3A%2F%2Fa.slack-edge.com%2Fdf10d%2Fimg%2Favatars%2Fava_0015-48.png", + "image_72": "https://secure.gravatar.com/avatar/f8cbaf456d5bed4fe78d758a5e8d27fc.jpg?s=72&d=https%3A%2F%2Fa.slack-edge.com%2Fdf10d%2Fimg%2Favatars%2Fava_0015-72.png", + "image_192": "https://secure.gravatar.com/avatar/f8cbaf456d5bed4fe78d758a5e8d27fc.jpg?s=192&d=https%3A%2F%2Fa.slack-edge.com%2Fdf10d%2Fimg%2Favatars%2Fava_0015-192.png", + "image_512": "https://secure.gravatar.com/avatar/f8cbaf456d5bed4fe78d758a5e8d27fc.jpg?s=512&d=https%3A%2F%2Fa.slack-edge.com%2Fdf10d%2Fimg%2Favatars%2Fava_0015-512.png", + "status_text_canonical": "", + "team": "T01CWUYP5AR" + }, + "is_admin": false, + "is_owner": false, + "is_primary_owner": false, + "is_restricted": false, + "is_ultra_restricted": false, + "is_bot": false, + "is_app_user": false, + "updated": 1684515104, + "is_email_confirmed": true, + "has_2fa": false, + "who_can_share_contact_card": "EVERYONE" + }, + "U05HBLE7YPL": { + "id": "U05HBLE7YPL", + "team_id": "T01CWUYP5AR", + "name": "abdallah", + "deleted": false, + "color": "385a86", + "real_name": "Abdallah", + "tz": "Europe/Amsterdam", + "tz_label": "Central European Time", + "tz_offset": 3600, + "profile": { + "title": "Staff Data Engineer @ Decathlon", + "phone": "", + "skype": "", + "real_name": "Abdallah", + "real_name_normalized": "Abdallah", + "display_name": "Abdallah", + "display_name_normalized": "Abdallah", + "fields": { + "Xf03UL3S8CP8": { + "value": "Staff Data Engineer @ Decathlon", + "alt": "" + } + }, + "status_text": "", + "status_emoji": "", + "status_emoji_display_info": [], + "status_expiration": 0, + "avatar_hash": "7373e3821b00", + "image_original": "https://avatars.slack-edge.com/2023-08-17/5743553612391_7373e3821b00a7631bfb_original.png", + "is_custom_image": true, + "email": "abdallah@terrab.me", + "first_name": "Abdallah", + "last_name": "", + "image_24": "https://avatars.slack-edge.com/2023-08-17/5743553612391_7373e3821b00a7631bfb_24.png", + "image_32": "https://avatars.slack-edge.com/2023-08-17/5743553612391_7373e3821b00a7631bfb_32.png", + "image_48": "https://avatars.slack-edge.com/2023-08-17/5743553612391_7373e3821b00a7631bfb_48.png", + "image_72": "https://avatars.slack-edge.com/2023-08-17/5743553612391_7373e3821b00a7631bfb_72.png", + "image_192": "https://avatars.slack-edge.com/2023-08-17/5743553612391_7373e3821b00a7631bfb_192.png", + "image_512": "https://avatars.slack-edge.com/2023-08-17/5743553612391_7373e3821b00a7631bfb_512.png", + "image_1024": "https://avatars.slack-edge.com/2023-08-17/5743553612391_7373e3821b00a7631bfb_1024.png", + "status_text_canonical": "", + "team": "T01CWUYP5AR" + }, + "is_admin": false, + "is_owner": false, + "is_primary_owner": false, + "is_restricted": false, + "is_ultra_restricted": false, + "is_bot": false, + "is_app_user": false, + "updated": 1700253819, + "is_email_confirmed": true, + "has_2fa": false, + "who_can_share_contact_card": "EVERYONE" + }, + "U0323HG8C8H": { + "id": "U0323HG8C8H", + "team_id": "T01CWUYP5AR", + "name": "sheeri.cabral", + "deleted": false, + "color": "2b6836", + "real_name": "Sheeri Cabral", + "tz": "America/New_York", + "tz_label": "Eastern Standard Time", + "tz_offset": -18000, + "profile": { + "title": "", + "phone": "", + "skype": "", + "real_name": "Sheeri Cabral", + "real_name_normalized": "Sheeri Cabral", + "display_name": "Sheeri Cabral (Collibra)", + "display_name_normalized": "Sheeri Cabral (Collibra)", + "fields": {}, + "status_text": "", + "status_emoji": "", + "status_emoji_display_info": [], + "status_expiration": 0, + "avatar_hash": "g0f9e11ecf9b", + "email": "sheeri.cabral@collibra.com", + "huddle_state": "default_unset", + "huddle_state_expiration_ts": 1673062422, + "first_name": "Sheeri", + "last_name": "Cabral", + "image_24": "https://secure.gravatar.com/avatar/0f9e11ecf9b45c68ee6f9d72f2b931d8.jpg?s=24&d=https%3A%2F%2Fa.slack-edge.com%2Fdf10d%2Fimg%2Favatars%2Fava_0006-24.png", + "image_32": "https://secure.gravatar.com/avatar/0f9e11ecf9b45c68ee6f9d72f2b931d8.jpg?s=32&d=https%3A%2F%2Fa.slack-edge.com%2Fdf10d%2Fimg%2Favatars%2Fava_0006-32.png", + "image_48": "https://secure.gravatar.com/avatar/0f9e11ecf9b45c68ee6f9d72f2b931d8.jpg?s=48&d=https%3A%2F%2Fa.slack-edge.com%2Fdf10d%2Fimg%2Favatars%2Fava_0006-48.png", + "image_72": "https://secure.gravatar.com/avatar/0f9e11ecf9b45c68ee6f9d72f2b931d8.jpg?s=72&d=https%3A%2F%2Fa.slack-edge.com%2Fdf10d%2Fimg%2Favatars%2Fava_0006-72.png", + "image_192": "https://secure.gravatar.com/avatar/0f9e11ecf9b45c68ee6f9d72f2b931d8.jpg?s=192&d=https%3A%2F%2Fa.slack-edge.com%2Fdf10d%2Fimg%2Favatars%2Fava_0006-192.png", + "image_512": "https://secure.gravatar.com/avatar/0f9e11ecf9b45c68ee6f9d72f2b931d8.jpg?s=512&d=https%3A%2F%2Fa.slack-edge.com%2Fdf10d%2Fimg%2Favatars%2Fava_0006-512.png", + "status_text_canonical": "", + "team": "T01CWUYP5AR" + }, + "is_admin": false, + "is_owner": false, + "is_primary_owner": false, + "is_restricted": false, + "is_ultra_restricted": false, + "is_bot": false, + "is_app_user": false, + "updated": 1678292994, + "is_email_confirmed": true, + "has_2fa": false, + "who_can_share_contact_card": "EVERYONE" + }, + "U05NGJ8AM8X": { + "id": "U05NGJ8AM8X", + "team_id": "T01CWUYP5AR", + "name": "yunhe52203334", + "deleted": false, + "color": "c386df", + "real_name": "Yunhe", + "tz": "Asia/Chongqing", + "tz_label": "China Standard Time", + "tz_offset": 28800, + "profile": { + "title": "", + "phone": "", + "skype": "", + "real_name": "Yunhe", + "real_name_normalized": "Yunhe", + "display_name": "Yunhe", + "display_name_normalized": "Yunhe", + "fields": {}, + "status_text": "", + "status_emoji": "", + "status_emoji_display_info": [], + "status_expiration": 0, + "avatar_hash": "g58baad279ee", + "email": "yunhe52203334@outlook.com", + "first_name": "Yunhe", + "last_name": "", + "image_24": "https://secure.gravatar.com/avatar/58baad279ee786786c00648fb1ad2cf6.jpg?s=24&d=https%3A%2F%2Fa.slack-edge.com%2Fdf10d%2Fimg%2Favatars%2Fava_0016-24.png", + "image_32": "https://secure.gravatar.com/avatar/58baad279ee786786c00648fb1ad2cf6.jpg?s=32&d=https%3A%2F%2Fa.slack-edge.com%2Fdf10d%2Fimg%2Favatars%2Fava_0016-32.png", + "image_48": "https://secure.gravatar.com/avatar/58baad279ee786786c00648fb1ad2cf6.jpg?s=48&d=https%3A%2F%2Fa.slack-edge.com%2Fdf10d%2Fimg%2Favatars%2Fava_0016-48.png", + "image_72": "https://secure.gravatar.com/avatar/58baad279ee786786c00648fb1ad2cf6.jpg?s=72&d=https%3A%2F%2Fa.slack-edge.com%2Fdf10d%2Fimg%2Favatars%2Fava_0016-72.png", + "image_192": "https://secure.gravatar.com/avatar/58baad279ee786786c00648fb1ad2cf6.jpg?s=192&d=https%3A%2F%2Fa.slack-edge.com%2Fdf10d%2Fimg%2Favatars%2Fava_0016-192.png", + "image_512": "https://secure.gravatar.com/avatar/58baad279ee786786c00648fb1ad2cf6.jpg?s=512&d=https%3A%2F%2Fa.slack-edge.com%2Fdf10d%2Fimg%2Favatars%2Fava_0016-512.png", + "status_text_canonical": "", + "team": "T01CWUYP5AR" + }, + "is_admin": false, + "is_owner": false, + "is_primary_owner": false, + "is_restricted": false, + "is_ultra_restricted": false, + "is_bot": false, + "is_app_user": false, + "updated": 1692702417, + "is_email_confirmed": true, + "has_2fa": false, + "who_can_share_contact_card": "EVERYONE" + }, + "U05EC8WB74N": { + "id": "U05EC8WB74N", + "team_id": "T01CWUYP5AR", + "name": "mbarrien", + "deleted": false, + "color": "7d414c", + "real_name": "Michael Barrientos", + "tz": "America/Los_Angeles", + "tz_label": "Pacific Standard Time", + "tz_offset": -28800, + "profile": { + "title": "", + "phone": "", + "skype": "", + "real_name": "Michael Barrientos", + "real_name_normalized": "Michael Barrientos", + "display_name": "", + "display_name_normalized": "", + "fields": {}, + "status_text": "", + "status_emoji": "", + "status_emoji_display_info": [], + "status_expiration": 0, + "avatar_hash": "ga99207eb377", + "email": "mbarrien@gmail.com", + "first_name": "Michael", + "last_name": "Barrientos", + "image_24": "https://secure.gravatar.com/avatar/a99207eb3777b2015dfd857f865b3376.jpg?s=24&d=https%3A%2F%2Fa.slack-edge.com%2Fdf10d%2Fimg%2Favatars%2Fava_0026-24.png", + "image_32": "https://secure.gravatar.com/avatar/a99207eb3777b2015dfd857f865b3376.jpg?s=32&d=https%3A%2F%2Fa.slack-edge.com%2Fdf10d%2Fimg%2Favatars%2Fava_0026-32.png", + "image_48": "https://secure.gravatar.com/avatar/a99207eb3777b2015dfd857f865b3376.jpg?s=48&d=https%3A%2F%2Fa.slack-edge.com%2Fdf10d%2Fimg%2Favatars%2Fava_0026-48.png", + "image_72": "https://secure.gravatar.com/avatar/a99207eb3777b2015dfd857f865b3376.jpg?s=72&d=https%3A%2F%2Fa.slack-edge.com%2Fdf10d%2Fimg%2Favatars%2Fava_0026-72.png", + "image_192": "https://secure.gravatar.com/avatar/a99207eb3777b2015dfd857f865b3376.jpg?s=192&d=https%3A%2F%2Fa.slack-edge.com%2Fdf10d%2Fimg%2Favatars%2Fava_0026-192.png", + "image_512": "https://secure.gravatar.com/avatar/a99207eb3777b2015dfd857f865b3376.jpg?s=512&d=https%3A%2F%2Fa.slack-edge.com%2Fdf10d%2Fimg%2Favatars%2Fava_0026-512.png", + "status_text_canonical": "", + "team": "T01CWUYP5AR" + }, + "is_admin": false, + "is_owner": false, + "is_primary_owner": false, + "is_restricted": false, + "is_ultra_restricted": false, + "is_bot": false, + "is_app_user": false, + "updated": 1689066386, + "is_email_confirmed": true, + "has_2fa": false, + "who_can_share_contact_card": "EVERYONE" + }, + "U05JY6MN8MS": { + "id": "U05JY6MN8MS", + "team_id": "T01CWUYP5AR", + "name": "githubopenlineageissu", + "deleted": false, + "color": "e475df", + "real_name": "GitHubOpenLineageIssues", + "tz": "America/New_York", + "tz_label": "Eastern Standard Time", + "tz_offset": -18000, + "profile": { + "title": "", + "phone": "", + "skype": "", + "real_name": "GitHubOpenLineageIssues", + "real_name_normalized": "GitHubOpenLineageIssues", + "display_name": "GitHubOpenLineageIssues", + "display_name_normalized": "GitHubOpenLineageIssues", + "fields": {}, + "status_text": "", + "status_emoji": "", + "status_emoji_display_info": [], + "status_expiration": 0, + "avatar_hash": "c3fd2e7f9e03", + "image_original": "https://avatars.slack-edge.com/2023-07-26/5638847371541_c3fd2e7f9e039ff2989e_original.png", + "is_custom_image": true, + "email": "githubopenlineageissues@gmail.com", + "first_name": "GitHubOpenLineageIssues", + "last_name": "", + "image_24": "https://avatars.slack-edge.com/2023-07-26/5638847371541_c3fd2e7f9e039ff2989e_24.png", + "image_32": "https://avatars.slack-edge.com/2023-07-26/5638847371541_c3fd2e7f9e039ff2989e_32.png", + "image_48": "https://avatars.slack-edge.com/2023-07-26/5638847371541_c3fd2e7f9e039ff2989e_48.png", + "image_72": "https://avatars.slack-edge.com/2023-07-26/5638847371541_c3fd2e7f9e039ff2989e_72.png", + "image_192": "https://avatars.slack-edge.com/2023-07-26/5638847371541_c3fd2e7f9e039ff2989e_192.png", + "image_512": "https://avatars.slack-edge.com/2023-07-26/5638847371541_c3fd2e7f9e039ff2989e_512.png", + "image_1024": "https://avatars.slack-edge.com/2023-07-26/5638847371541_c3fd2e7f9e039ff2989e_1024.png", + "status_text_canonical": "", + "team": "T01CWUYP5AR" + }, + "is_admin": false, + "is_owner": false, + "is_primary_owner": false, + "is_restricted": false, + "is_ultra_restricted": false, + "is_bot": false, + "is_app_user": false, + "updated": 1690382661, + "is_email_confirmed": true, + "has_2fa": false, + "who_can_share_contact_card": "EVERYONE" + }, + "U05PVS8GRJ6": { + "id": "U05PVS8GRJ6", + "team_id": "T01CWUYP5AR", + "name": "josdotso", + "deleted": false, + "color": "902d59", + "real_name": "Joshua Dotson", + "tz": "America/New_York", + "tz_label": "Eastern Standard Time", + "tz_offset": -18000, + "profile": { + "title": "", + "phone": "", + "skype": "", + "real_name": "Joshua Dotson", + "real_name_normalized": "Joshua Dotson", + "display_name": "Joshua Dotson", + "display_name_normalized": "Joshua Dotson", + "fields": { + "Xf01NDQXPW6A": { + "value": "https://github.com/josdotso", + "alt": "@GitHub" + } + }, + "status_text": "", + "status_emoji": "", + "status_emoji_display_info": [], + "status_expiration": 0, + "avatar_hash": "gf5f77a7979d", + "email": "josdotso@cisco.com", + "first_name": "Joshua", + "last_name": "Dotson", + "image_24": "https://secure.gravatar.com/avatar/f5f77a7979d0cec8f762e393eeb5a010.jpg?s=24&d=https%3A%2F%2Fa.slack-edge.com%2Fdf10d%2Fimg%2Favatars%2Fava_0026-24.png", + "image_32": "https://secure.gravatar.com/avatar/f5f77a7979d0cec8f762e393eeb5a010.jpg?s=32&d=https%3A%2F%2Fa.slack-edge.com%2Fdf10d%2Fimg%2Favatars%2Fava_0026-32.png", + "image_48": "https://secure.gravatar.com/avatar/f5f77a7979d0cec8f762e393eeb5a010.jpg?s=48&d=https%3A%2F%2Fa.slack-edge.com%2Fdf10d%2Fimg%2Favatars%2Fava_0026-48.png", + "image_72": "https://secure.gravatar.com/avatar/f5f77a7979d0cec8f762e393eeb5a010.jpg?s=72&d=https%3A%2F%2Fa.slack-edge.com%2Fdf10d%2Fimg%2Favatars%2Fava_0026-72.png", + "image_192": "https://secure.gravatar.com/avatar/f5f77a7979d0cec8f762e393eeb5a010.jpg?s=192&d=https%3A%2F%2Fa.slack-edge.com%2Fdf10d%2Fimg%2Favatars%2Fava_0026-192.png", + "image_512": "https://secure.gravatar.com/avatar/f5f77a7979d0cec8f762e393eeb5a010.jpg?s=512&d=https%3A%2F%2Fa.slack-edge.com%2Fdf10d%2Fimg%2Favatars%2Fava_0026-512.png", + "status_text_canonical": "", + "team": "T01CWUYP5AR" + }, + "is_admin": false, + "is_owner": false, + "is_primary_owner": false, + "is_restricted": false, + "is_ultra_restricted": false, + "is_bot": false, + "is_app_user": false, + "updated": 1693243608, + "is_email_confirmed": true, + "has_2fa": false, + "who_can_share_contact_card": "EVERYONE" + }, + "U05J5GRKY10": { + "id": "U05J5GRKY10", + "team_id": "T01CWUYP5AR", + "name": "george.polychronopoul", + "deleted": false, + "color": "8d4b84", + "real_name": "George Polychronopoulos", + "tz": "Europe/London", + "tz_label": "Greenwich Mean Time", + "tz_offset": 0, + "profile": { + "title": "", + "phone": "", + "skype": "", + "real_name": "George Polychronopoulos", + "real_name_normalized": "George Polychronopoulos", + "display_name": "George Polychronopoulos", + "display_name_normalized": "George Polychronopoulos", + "fields": {}, + "status_text": "", + "status_emoji": "", + "status_emoji_display_info": [], + "status_expiration": 0, + "avatar_hash": "378e4aaddd2e", + "image_original": "https://avatars.slack-edge.com/2023-09-04/5870018949296_378e4aaddd2ed569ad37_original.png", + "is_custom_image": true, + "email": "george.polychronopoulos@6point6.co.uk", + "huddle_state": "default_unset", + "huddle_state_expiration_ts": 0, + "first_name": "George", + "last_name": "Polychronopoulos", + "image_24": "https://avatars.slack-edge.com/2023-09-04/5870018949296_378e4aaddd2ed569ad37_24.png", + "image_32": "https://avatars.slack-edge.com/2023-09-04/5870018949296_378e4aaddd2ed569ad37_32.png", + "image_48": "https://avatars.slack-edge.com/2023-09-04/5870018949296_378e4aaddd2ed569ad37_48.png", + "image_72": "https://avatars.slack-edge.com/2023-09-04/5870018949296_378e4aaddd2ed569ad37_72.png", + "image_192": "https://avatars.slack-edge.com/2023-09-04/5870018949296_378e4aaddd2ed569ad37_192.png", + "image_512": "https://avatars.slack-edge.com/2023-09-04/5870018949296_378e4aaddd2ed569ad37_512.png", + "image_1024": "https://avatars.slack-edge.com/2023-09-04/5870018949296_378e4aaddd2ed569ad37_1024.png", + "status_text_canonical": "", + "team": "T01CWUYP5AR" + }, + "is_admin": false, + "is_owner": false, + "is_primary_owner": false, + "is_restricted": false, + "is_ultra_restricted": false, + "is_bot": false, + "is_app_user": false, + "updated": 1693935437, + "is_email_confirmed": true, + "has_2fa": false, + "who_can_share_contact_card": "EVERYONE" + }, + "U05TQPZ4R4L": { + "id": "U05TQPZ4R4L", + "team_id": "T01CWUYP5AR", + "name": "aaruna6", + "deleted": false, + "color": "e0a729", + "real_name": "Aaruna Godthi", + "tz": "America/Los_Angeles", + "tz_label": "Pacific Standard Time", + "tz_offset": -28800, + "profile": { + "title": "", + "phone": "", + "skype": "", + "real_name": "Aaruna Godthi", + "real_name_normalized": "Aaruna Godthi", + "display_name": "Aaruna Godthi", + "display_name_normalized": "Aaruna Godthi", + "fields": {}, + "status_text": "", + "status_emoji": "", + "status_emoji_display_info": [], + "status_expiration": 0, + "avatar_hash": "fea59103aed5", + "image_original": "https://avatars.slack-edge.com/2023-09-23/5953004023841_fea59103aed522db7cba_original.png", + "is_custom_image": true, + "email": "aaruna6@gmail.com", + "first_name": "Aaruna", + "last_name": "Godthi", + "image_24": "https://avatars.slack-edge.com/2023-09-23/5953004023841_fea59103aed522db7cba_24.png", + "image_32": "https://avatars.slack-edge.com/2023-09-23/5953004023841_fea59103aed522db7cba_32.png", + "image_48": "https://avatars.slack-edge.com/2023-09-23/5953004023841_fea59103aed522db7cba_48.png", + "image_72": "https://avatars.slack-edge.com/2023-09-23/5953004023841_fea59103aed522db7cba_72.png", + "image_192": "https://avatars.slack-edge.com/2023-09-23/5953004023841_fea59103aed522db7cba_192.png", + "image_512": "https://avatars.slack-edge.com/2023-09-23/5953004023841_fea59103aed522db7cba_512.png", + "image_1024": "https://avatars.slack-edge.com/2023-09-23/5953004023841_fea59103aed522db7cba_1024.png", + "status_text_canonical": "", + "team": "T01CWUYP5AR" + }, + "is_admin": false, + "is_owner": false, + "is_primary_owner": false, + "is_restricted": false, + "is_ultra_restricted": false, + "is_bot": false, + "is_app_user": false, + "updated": 1695501948, + "is_email_confirmed": true, + "has_2fa": false, + "who_can_share_contact_card": "EVERYONE" + }, + "U05Q3HT6PBR": { + "id": "U05Q3HT6PBR", + "team_id": "T01CWUYP5AR", + "name": "kevin", + "deleted": false, + "color": "e23f99", + "real_name": "Kevin Languasco", + "tz": "America/Bogota", + "tz_label": "South America Pacific Standard Time", + "tz_offset": -18000, + "profile": { + "title": "", + "phone": "", + "skype": "", + "real_name": "Kevin Languasco", + "real_name_normalized": "Kevin Languasco", + "display_name": "Kevin Languasco", + "display_name_normalized": "Kevin Languasco", + "fields": {}, + "status_text": "", + "status_emoji": "", + "status_emoji_display_info": [], + "status_expiration": 0, + "avatar_hash": "adc56d6bedf9", + "image_original": "https://avatars.slack-edge.com/2023-08-31/5836587630708_adc56d6bedf9b9198ccf_original.jpg", + "is_custom_image": true, + "email": "kevin@haystack.tv", + "first_name": "Kevin", + "last_name": "Languasco", + "image_24": "https://avatars.slack-edge.com/2023-08-31/5836587630708_adc56d6bedf9b9198ccf_24.jpg", + "image_32": "https://avatars.slack-edge.com/2023-08-31/5836587630708_adc56d6bedf9b9198ccf_32.jpg", + "image_48": "https://avatars.slack-edge.com/2023-08-31/5836587630708_adc56d6bedf9b9198ccf_48.jpg", + "image_72": "https://avatars.slack-edge.com/2023-08-31/5836587630708_adc56d6bedf9b9198ccf_72.jpg", + "image_192": "https://avatars.slack-edge.com/2023-08-31/5836587630708_adc56d6bedf9b9198ccf_192.jpg", + "image_512": "https://avatars.slack-edge.com/2023-08-31/5836587630708_adc56d6bedf9b9198ccf_512.jpg", + "image_1024": "https://avatars.slack-edge.com/2023-08-31/5836587630708_adc56d6bedf9b9198ccf_1024.jpg", + "status_text_canonical": "", + "team": "T01CWUYP5AR" + }, + "is_admin": false, + "is_owner": false, + "is_primary_owner": false, + "is_restricted": false, + "is_ultra_restricted": false, + "is_bot": false, + "is_app_user": false, + "updated": 1700259423, + "is_email_confirmed": true, + "has_2fa": false, + "who_can_share_contact_card": "EVERYONE" + }, + "U05SXDWVA7K": { + "id": "U05SXDWVA7K", + "team_id": "T01CWUYP5AR", + "name": "kgkwiz", + "deleted": false, + "color": "e85d72", + "real_name": "Greg Kim", + "tz": "America/New_York", + "tz_label": "Eastern Standard Time", + "tz_offset": -18000, + "profile": { + "title": "", + "phone": "", + "skype": "", + "real_name": "Greg Kim", + "real_name_normalized": "Greg Kim", + "display_name": "Greg Kim", + "display_name_normalized": "Greg Kim", + "fields": {}, + "status_text": "", + "status_emoji": "", + "status_emoji_display_info": [], + "status_expiration": 0, + "avatar_hash": "8c0cc5d026c2", + "image_original": "https://avatars.slack-edge.com/2023-09-15/5905283953172_8c0cc5d026c289ba0f06_original.png", + "is_custom_image": true, + "email": "kgkwiz@gmail.com", + "first_name": "Greg", + "last_name": "Kim", + "image_24": "https://avatars.slack-edge.com/2023-09-15/5905283953172_8c0cc5d026c289ba0f06_24.png", + "image_32": "https://avatars.slack-edge.com/2023-09-15/5905283953172_8c0cc5d026c289ba0f06_32.png", + "image_48": "https://avatars.slack-edge.com/2023-09-15/5905283953172_8c0cc5d026c289ba0f06_48.png", + "image_72": "https://avatars.slack-edge.com/2023-09-15/5905283953172_8c0cc5d026c289ba0f06_72.png", + "image_192": "https://avatars.slack-edge.com/2023-09-15/5905283953172_8c0cc5d026c289ba0f06_192.png", + "image_512": "https://avatars.slack-edge.com/2023-09-15/5905283953172_8c0cc5d026c289ba0f06_512.png", + "image_1024": "https://avatars.slack-edge.com/2023-09-15/5905283953172_8c0cc5d026c289ba0f06_1024.png", + "status_text_canonical": "", + "team": "T01CWUYP5AR" + }, + "is_admin": false, + "is_owner": false, + "is_primary_owner": false, + "is_restricted": false, + "is_ultra_restricted": false, + "is_bot": false, + "is_app_user": false, + "updated": 1694786964, + "is_email_confirmed": true, + "has_2fa": false, + "who_can_share_contact_card": "EVERYONE" + }, + "U01HNKK4XAM": { + "id": "U01HNKK4XAM", + "team_id": "T01CWUYP5AR", + "name": "harel.shein", + "deleted": false, + "color": "d55aef", + "real_name": "Harel Shein", + "tz": "America/New_York", + "tz_label": "Eastern Standard Time", + "tz_offset": -18000, + "profile": { + "title": "Data Observability @ Astronomer", + "phone": "", + "skype": "", + "real_name": "Harel Shein", + "real_name_normalized": "Harel Shein", + "display_name": "Harel Shein", + "display_name_normalized": "Harel Shein", + "fields": { + "Xf03UL3S8CP8": { + "value": "Data Observability @ Astronomer", + "alt": "" + } + }, + "status_text": "", + "status_emoji": "", + "status_emoji_display_info": [], + "status_expiration": 0, + "avatar_hash": "7467955ae073", + "image_original": "https://avatars.slack-edge.com/2020-12-18/1581303887318_7467955ae073e3a3757f_original.jpg", + "is_custom_image": true, + "email": "harel.shein@gmail.com", + "image_24": "https://avatars.slack-edge.com/2020-12-18/1581303887318_7467955ae073e3a3757f_24.jpg", + "image_32": "https://avatars.slack-edge.com/2020-12-18/1581303887318_7467955ae073e3a3757f_32.jpg", + "image_48": "https://avatars.slack-edge.com/2020-12-18/1581303887318_7467955ae073e3a3757f_48.jpg", + "image_72": "https://avatars.slack-edge.com/2020-12-18/1581303887318_7467955ae073e3a3757f_72.jpg", + "image_192": "https://avatars.slack-edge.com/2020-12-18/1581303887318_7467955ae073e3a3757f_192.jpg", + "image_512": "https://avatars.slack-edge.com/2020-12-18/1581303887318_7467955ae073e3a3757f_512.jpg", + "image_1024": "https://avatars.slack-edge.com/2020-12-18/1581303887318_7467955ae073e3a3757f_1024.jpg", + "status_text_canonical": "", + "team": "T01CWUYP5AR" + }, + "is_admin": true, + "is_owner": false, + "is_primary_owner": false, + "is_restricted": false, + "is_ultra_restricted": false, + "is_bot": false, + "is_app_user": false, + "updated": 1700085172, + "is_email_confirmed": true, + "has_2fa": false, + "who_can_share_contact_card": "EVERYONE" + }, + "U05QHG1NJ8J": { + "id": "U05QHG1NJ8J", + "team_id": "T01CWUYP5AR", + "name": "mike474", + "deleted": false, + "color": "4ec0d6", + "real_name": "Mike O'Connor", + "tz": "Europe/London", + "tz_label": "Greenwich Mean Time", + "tz_offset": 0, + "profile": { + "title": "", + "phone": "", + "skype": "", + "real_name": "Mike O'Connor", + "real_name_normalized": "Mike O'Connor", + "display_name": "Mike O'Connor", + "display_name_normalized": "Mike O'Connor", + "fields": {}, + "status_text": "", + "status_emoji": "", + "status_emoji_display_info": [], + "status_expiration": 0, + "avatar_hash": "gdc94937fa8a", + "email": "mike@entos.ai", + "first_name": "Mike", + "last_name": "O'Connor", + "image_24": "https://secure.gravatar.com/avatar/dc94937fa8a4517c54848b990c54fbbe.jpg?s=24&d=https%3A%2F%2Fa.slack-edge.com%2Fdf10d%2Fimg%2Favatars%2Fava_0003-24.png", + "image_32": "https://secure.gravatar.com/avatar/dc94937fa8a4517c54848b990c54fbbe.jpg?s=32&d=https%3A%2F%2Fa.slack-edge.com%2Fdf10d%2Fimg%2Favatars%2Fava_0003-32.png", + "image_48": "https://secure.gravatar.com/avatar/dc94937fa8a4517c54848b990c54fbbe.jpg?s=48&d=https%3A%2F%2Fa.slack-edge.com%2Fdf10d%2Fimg%2Favatars%2Fava_0003-48.png", + "image_72": "https://secure.gravatar.com/avatar/dc94937fa8a4517c54848b990c54fbbe.jpg?s=72&d=https%3A%2F%2Fa.slack-edge.com%2Fdf10d%2Fimg%2Favatars%2Fava_0003-72.png", + "image_192": "https://secure.gravatar.com/avatar/dc94937fa8a4517c54848b990c54fbbe.jpg?s=192&d=https%3A%2F%2Fa.slack-edge.com%2Fdf10d%2Fimg%2Favatars%2Fava_0003-192.png", + "image_512": "https://secure.gravatar.com/avatar/dc94937fa8a4517c54848b990c54fbbe.jpg?s=512&d=https%3A%2F%2Fa.slack-edge.com%2Fdf10d%2Fimg%2Favatars%2Fava_0003-512.png", + "status_text_canonical": "", + "team": "T01CWUYP5AR" + }, + "is_admin": false, + "is_owner": false, + "is_primary_owner": false, + "is_restricted": false, + "is_ultra_restricted": false, + "is_bot": false, + "is_app_user": false, + "updated": 1693512222, + "is_email_confirmed": true, + "has_2fa": false, + "who_can_share_contact_card": "EVERYONE" + }, + "U01RA9B5GG2": { + "id": "U01RA9B5GG2", + "team_id": "T01CWUYP5AR", + "name": "maciej.obuchowski", + "deleted": false, + "color": "385a86", + "real_name": "Maciej Obuchowski", + "tz": "Europe/Warsaw", + "tz_label": "Central European Time", + "tz_offset": 3600, + "profile": { + "title": "OpenLineage committer", + "phone": "", + "skype": "", + "real_name": "Maciej Obuchowski", + "real_name_normalized": "Maciej Obuchowski", + "display_name": "Maciej Obuchowski", + "display_name_normalized": "Maciej Obuchowski", + "fields": {}, + "status_text": "", + "status_emoji": "", + "status_emoji_display_info": [], + "status_expiration": 0, + "avatar_hash": "2f556751039c", + "image_original": "https://avatars.slack-edge.com/2021-03-16/1888464110336_2f556751039c2a20f1fe_original.jpg", + "is_custom_image": true, + "email": "maciej.obuchowski@getindata.com", + "huddle_state": "default_unset", + "huddle_state_expiration_ts": 0, + "first_name": "Maciej", + "last_name": "Obuchowski", + "image_24": "https://avatars.slack-edge.com/2021-03-16/1888464110336_2f556751039c2a20f1fe_24.jpg", + "image_32": "https://avatars.slack-edge.com/2021-03-16/1888464110336_2f556751039c2a20f1fe_32.jpg", + "image_48": "https://avatars.slack-edge.com/2021-03-16/1888464110336_2f556751039c2a20f1fe_48.jpg", + "image_72": "https://avatars.slack-edge.com/2021-03-16/1888464110336_2f556751039c2a20f1fe_72.jpg", + "image_192": "https://avatars.slack-edge.com/2021-03-16/1888464110336_2f556751039c2a20f1fe_192.jpg", + "image_512": "https://avatars.slack-edge.com/2021-03-16/1888464110336_2f556751039c2a20f1fe_512.jpg", + "image_1024": "https://avatars.slack-edge.com/2021-03-16/1888464110336_2f556751039c2a20f1fe_1024.jpg", + "status_text_canonical": "", + "team": "T01CWUYP5AR" + }, + "is_admin": false, + "is_owner": false, + "is_primary_owner": false, + "is_restricted": false, + "is_ultra_restricted": false, + "is_bot": false, + "is_app_user": false, + "updated": 1698879550, + "is_email_confirmed": true, + "has_2fa": false, + "who_can_share_contact_card": "EVERYONE" + }, + "U0620HU51HA": { + "id": "U0620HU51HA", + "team_id": "T01CWUYP5AR", + "name": "sicotte.jason", + "deleted": false, + "color": "5870dd", + "real_name": "Jason", + "tz": "America/Chicago", + "tz_label": "Central Standard Time", + "tz_offset": -21600, + "profile": { + "title": "", + "phone": "", + "skype": "", + "real_name": "Jason", + "real_name_normalized": "Jason", + "display_name": "Jason", + "display_name_normalized": "Jason", + "fields": {}, + "status_text": "", + "status_emoji": "", + "status_emoji_display_info": [], + "status_expiration": 0, + "avatar_hash": "g490f5667d26", + "email": "sicotte.jason@gmail.com", + "first_name": "Jason", + "last_name": "", + "image_24": "https://secure.gravatar.com/avatar/490f5667d260b49c54dde5b4d0fee938.jpg?s=24&d=https%3A%2F%2Fa.slack-edge.com%2Fdf10d%2Fimg%2Favatars%2Fava_0003-24.png", + "image_32": "https://secure.gravatar.com/avatar/490f5667d260b49c54dde5b4d0fee938.jpg?s=32&d=https%3A%2F%2Fa.slack-edge.com%2Fdf10d%2Fimg%2Favatars%2Fava_0003-32.png", + "image_48": "https://secure.gravatar.com/avatar/490f5667d260b49c54dde5b4d0fee938.jpg?s=48&d=https%3A%2F%2Fa.slack-edge.com%2Fdf10d%2Fimg%2Favatars%2Fava_0003-48.png", + "image_72": "https://secure.gravatar.com/avatar/490f5667d260b49c54dde5b4d0fee938.jpg?s=72&d=https%3A%2F%2Fa.slack-edge.com%2Fdf10d%2Fimg%2Favatars%2Fava_0003-72.png", + "image_192": "https://secure.gravatar.com/avatar/490f5667d260b49c54dde5b4d0fee938.jpg?s=192&d=https%3A%2F%2Fa.slack-edge.com%2Fdf10d%2Fimg%2Favatars%2Fava_0003-192.png", + "image_512": "https://secure.gravatar.com/avatar/490f5667d260b49c54dde5b4d0fee938.jpg?s=512&d=https%3A%2F%2Fa.slack-edge.com%2Fdf10d%2Fimg%2Favatars%2Fava_0003-512.png", + "status_text_canonical": "", + "team": "T01CWUYP5AR" + }, + "is_admin": false, + "is_owner": false, + "is_primary_owner": false, + "is_restricted": false, + "is_ultra_restricted": false, + "is_bot": false, + "is_app_user": false, + "updated": 1697734908, + "is_email_confirmed": true, + "has_2fa": false, + "who_can_share_contact_card": "EVERYONE" + }, + "U05U9K21LSG": { + "id": "U05U9K21LSG", + "team_id": "T01CWUYP5AR", + "name": "bill", + "deleted": false, + "color": "df3dc0", + "real_name": "Bill Dirks", + "tz": "America/Los_Angeles", + "tz_label": "Pacific Standard Time", + "tz_offset": -28800, + "profile": { + "title": "", + "phone": "", + "skype": "", + "real_name": "Bill Dirks", + "real_name_normalized": "Bill Dirks", + "display_name": "Bill Dirks", + "display_name_normalized": "Bill Dirks", + "fields": {}, + "status_text": "", + "status_emoji": "", + "status_emoji_display_info": [], + "status_expiration": 0, + "avatar_hash": "4d72939742b4", + "image_original": "https://avatars.slack-edge.com/2023-09-27/5959100825619_4d72939742b4643f68e6_original.jpg", + "is_custom_image": true, + "email": "bill@greatexpectations.io", + "first_name": "Bill", + "last_name": "Dirks", + "image_24": "https://avatars.slack-edge.com/2023-09-27/5959100825619_4d72939742b4643f68e6_24.jpg", + "image_32": "https://avatars.slack-edge.com/2023-09-27/5959100825619_4d72939742b4643f68e6_32.jpg", + "image_48": "https://avatars.slack-edge.com/2023-09-27/5959100825619_4d72939742b4643f68e6_48.jpg", + "image_72": "https://avatars.slack-edge.com/2023-09-27/5959100825619_4d72939742b4643f68e6_72.jpg", + "image_192": "https://avatars.slack-edge.com/2023-09-27/5959100825619_4d72939742b4643f68e6_192.jpg", + "image_512": "https://avatars.slack-edge.com/2023-09-27/5959100825619_4d72939742b4643f68e6_512.jpg", + "image_1024": "https://avatars.slack-edge.com/2023-09-27/5959100825619_4d72939742b4643f68e6_1024.jpg", + "status_text_canonical": "", + "team": "T01CWUYP5AR" + }, + "is_admin": false, + "is_owner": false, + "is_primary_owner": false, + "is_restricted": false, + "is_ultra_restricted": false, + "is_bot": false, + "is_app_user": false, + "updated": 1695839552, + "is_email_confirmed": true, + "has_2fa": false, + "who_can_share_contact_card": "EVERYONE" + }, + "U02S6F54MAB": { + "id": "U02S6F54MAB", + "team_id": "T01CWUYP5AR", + "name": "jakub.dardzinski", + "deleted": false, + "color": "e06b56", + "real_name": "Jakub Dardziński", + "tz": "Europe/Warsaw", + "tz_label": "Central European Time", + "tz_offset": 3600, + "profile": { + "title": "", + "phone": "", + "skype": "", + "real_name": "Jakub Dardziński", + "real_name_normalized": "Jakub Dardzinski", + "display_name": "Jakub Dardziński", + "display_name_normalized": "Jakub Dardzinski", + "fields": {}, + "status_text": "", + "status_emoji": "", + "status_emoji_display_info": [], + "status_expiration": 0, + "avatar_hash": "ba977a7f01d7", + "image_original": "https://avatars.slack-edge.com/2022-01-04/2902203539797_ba977a7f01d752dca1ca_original.jpg", + "is_custom_image": true, + "email": "jakub.dardzinski@getindata.com", + "first_name": "Jakub", + "last_name": "Dardziński", + "image_24": "https://avatars.slack-edge.com/2022-01-04/2902203539797_ba977a7f01d752dca1ca_24.jpg", + "image_32": "https://avatars.slack-edge.com/2022-01-04/2902203539797_ba977a7f01d752dca1ca_32.jpg", + "image_48": "https://avatars.slack-edge.com/2022-01-04/2902203539797_ba977a7f01d752dca1ca_48.jpg", + "image_72": "https://avatars.slack-edge.com/2022-01-04/2902203539797_ba977a7f01d752dca1ca_72.jpg", + "image_192": "https://avatars.slack-edge.com/2022-01-04/2902203539797_ba977a7f01d752dca1ca_192.jpg", + "image_512": "https://avatars.slack-edge.com/2022-01-04/2902203539797_ba977a7f01d752dca1ca_512.jpg", + "image_1024": "https://avatars.slack-edge.com/2022-01-04/2902203539797_ba977a7f01d752dca1ca_1024.jpg", + "status_text_canonical": "", + "team": "T01CWUYP5AR" + }, + "is_admin": false, + "is_owner": false, + "is_primary_owner": false, + "is_restricted": false, + "is_ultra_restricted": false, + "is_bot": false, + "is_app_user": false, + "updated": 1695527453, + "is_email_confirmed": true, + "has_2fa": false, + "who_can_share_contact_card": "EVERYONE" + }, + "U05U9929K3N": { + "id": "U05U9929K3N", + "team_id": "T01CWUYP5AR", + "name": "don", + "deleted": false, + "color": "99a949", + "real_name": "Don Heppner", + "tz": "America/New_York", + "tz_label": "Eastern Standard Time", + "tz_offset": -18000, + "profile": { + "title": "", + "phone": "", + "skype": "", + "real_name": "Don Heppner", + "real_name_normalized": "Don Heppner", + "display_name": "Don Heppner", + "display_name_normalized": "Don Heppner", + "fields": {}, + "status_text": "", + "status_emoji": "", + "status_emoji_display_info": [], + "status_expiration": 0, + "avatar_hash": "87e9efa4de64", + "image_original": "https://avatars.slack-edge.com/2023-09-27/5981853048288_87e9efa4de64acdbd3c7_original.png", + "is_custom_image": true, + "email": "don@greatexpectations.io", + "first_name": "Don", + "last_name": "Heppner", + "image_24": "https://avatars.slack-edge.com/2023-09-27/5981853048288_87e9efa4de64acdbd3c7_24.png", + "image_32": "https://avatars.slack-edge.com/2023-09-27/5981853048288_87e9efa4de64acdbd3c7_32.png", + "image_48": "https://avatars.slack-edge.com/2023-09-27/5981853048288_87e9efa4de64acdbd3c7_48.png", + "image_72": "https://avatars.slack-edge.com/2023-09-27/5981853048288_87e9efa4de64acdbd3c7_72.png", + "image_192": "https://avatars.slack-edge.com/2023-09-27/5981853048288_87e9efa4de64acdbd3c7_192.png", + "image_512": "https://avatars.slack-edge.com/2023-09-27/5981853048288_87e9efa4de64acdbd3c7_512.png", + "image_1024": "https://avatars.slack-edge.com/2023-09-27/5981853048288_87e9efa4de64acdbd3c7_1024.png", + "status_text_canonical": "", + "team": "T01CWUYP5AR" + }, + "is_admin": false, + "is_owner": false, + "is_primary_owner": false, + "is_restricted": false, + "is_ultra_restricted": false, + "is_bot": false, + "is_app_user": false, + "updated": 1697754119, + "is_email_confirmed": true, + "has_2fa": false, + "who_can_share_contact_card": "EVERYONE" + }, + "U01DCMDFHBK": { + "id": "U01DCMDFHBK", + "team_id": "T01CWUYP5AR", + "name": "willy", + "deleted": false, + "color": "df3dc0", + "real_name": "Willy Lulciuc", + "tz": "America/Los_Angeles", + "tz_label": "Pacific Standard Time", + "tz_offset": -28800, + "profile": { + "title": "Founding Engineer of Datakin, Co-creator Marquez, OpenLineage Committer", + "phone": "", + "skype": "", + "real_name": "Willy Lulciuc", + "real_name_normalized": "Willy Lulciuc", + "display_name": "Willy Lulciuc", + "display_name_normalized": "Willy Lulciuc", + "fields": { + "Xf01NSPCJ71S": { + "value": "willy@datakin.com", + "alt": "" + } + }, + "status_text": "", + "status_emoji": "", + "status_emoji_display_info": [], + "status_expiration": 0, + "avatar_hash": "cff4548fe7d7", + "image_original": "https://avatars.slack-edge.com/2020-10-21/1437812757414_cff4548fe7d7cd1b1744_original.jpg", + "is_custom_image": true, + "email": "willy@datakin.com", + "first_name": "Willy", + "last_name": "Lulciuc", + "image_24": "https://avatars.slack-edge.com/2020-10-21/1437812757414_cff4548fe7d7cd1b1744_24.jpg", + "image_32": "https://avatars.slack-edge.com/2020-10-21/1437812757414_cff4548fe7d7cd1b1744_32.jpg", + "image_48": "https://avatars.slack-edge.com/2020-10-21/1437812757414_cff4548fe7d7cd1b1744_48.jpg", + "image_72": "https://avatars.slack-edge.com/2020-10-21/1437812757414_cff4548fe7d7cd1b1744_72.jpg", + "image_192": "https://avatars.slack-edge.com/2020-10-21/1437812757414_cff4548fe7d7cd1b1744_192.jpg", + "image_512": "https://avatars.slack-edge.com/2020-10-21/1437812757414_cff4548fe7d7cd1b1744_512.jpg", + "image_1024": "https://avatars.slack-edge.com/2020-10-21/1437812757414_cff4548fe7d7cd1b1744_1024.jpg", + "status_text_canonical": "", + "team": "T01CWUYP5AR" + }, + "is_admin": true, + "is_owner": true, + "is_primary_owner": false, + "is_restricted": false, + "is_ultra_restricted": false, + "is_bot": false, + "is_app_user": false, + "updated": 1695661607, + "is_email_confirmed": true, + "has_2fa": false, + "who_can_share_contact_card": "EVERYONE" + }, + "U02MK6YNAQ5": { + "id": "U02MK6YNAQ5", + "team_id": "T01CWUYP5AR", + "name": "pawel.leszczynski", + "deleted": false, + "color": "a2a5dc", + "real_name": "Paweł Leszczyński", + "tz": "Europe/Amsterdam", + "tz_label": "Central European Time", + "tz_offset": 3600, + "profile": { + "title": "", + "phone": "", + "skype": "", + "real_name": "Paweł Leszczyński", + "real_name_normalized": "Pawel Leszczynski", + "display_name": "Paweł Leszczyński", + "display_name_normalized": "Pawel Leszczynski", + "fields": { + "Xf01NDQXPW6A": { + "value": "https://github.com/pawel-big-lebowski", + "alt": "" + } + }, + "status_text": "", + "status_emoji": "", + "status_emoji_display_info": [], + "status_expiration": 0, + "avatar_hash": "fcf3e1f61aa5", + "image_original": "https://avatars.slack-edge.com/2022-10-05/4175232328373_fcf3e1f61aa5625a2574_original.jpg", + "is_custom_image": true, + "email": "pawel.leszczynski@getindata.com", + "huddle_state": "default_unset", + "huddle_state_expiration_ts": 0, + "first_name": "Paweł", + "last_name": "Leszczyński", + "image_24": "https://avatars.slack-edge.com/2022-10-05/4175232328373_fcf3e1f61aa5625a2574_24.jpg", + "image_32": "https://avatars.slack-edge.com/2022-10-05/4175232328373_fcf3e1f61aa5625a2574_32.jpg", + "image_48": "https://avatars.slack-edge.com/2022-10-05/4175232328373_fcf3e1f61aa5625a2574_48.jpg", + "image_72": "https://avatars.slack-edge.com/2022-10-05/4175232328373_fcf3e1f61aa5625a2574_72.jpg", + "image_192": "https://avatars.slack-edge.com/2022-10-05/4175232328373_fcf3e1f61aa5625a2574_192.jpg", + "image_512": "https://avatars.slack-edge.com/2022-10-05/4175232328373_fcf3e1f61aa5625a2574_512.jpg", + "image_1024": "https://avatars.slack-edge.com/2022-10-05/4175232328373_fcf3e1f61aa5625a2574_1024.jpg", + "status_text_canonical": "", + "team": "T01CWUYP5AR" + }, + "is_admin": false, + "is_owner": false, + "is_primary_owner": false, + "is_restricted": false, + "is_ultra_restricted": false, + "is_bot": false, + "is_app_user": false, + "updated": 1700085153, + "is_email_confirmed": true, + "has_2fa": false, + "who_can_share_contact_card": "EVERYONE" + }, + "U053LLVTHRN": { + "id": "U053LLVTHRN", + "team_id": "T01CWUYP5AR", + "name": "ross769", + "deleted": false, + "color": "e475df", + "real_name": "Ross Turk", + "tz": "America/New_York", + "tz_label": "Eastern Standard Time", + "tz_offset": -18000, + "profile": { + "title": "", + "phone": "", + "skype": "", + "real_name": "Ross Turk", + "real_name_normalized": "Ross Turk", + "display_name": "", + "display_name_normalized": "", + "fields": {}, + "status_text": "", + "status_emoji": "", + "status_emoji_display_info": [], + "status_expiration": 0, + "avatar_hash": "g071531b6d09", + "email": "ross@rossturk.com", + "first_name": "Ross", + "last_name": "Turk", + "image_24": "https://secure.gravatar.com/avatar/071531b6d095f27a7cf6389dcfbb5766.jpg?s=24&d=https%3A%2F%2Fa.slack-edge.com%2Fdf10d%2Fimg%2Favatars%2Fava_0025-24.png", + "image_32": "https://secure.gravatar.com/avatar/071531b6d095f27a7cf6389dcfbb5766.jpg?s=32&d=https%3A%2F%2Fa.slack-edge.com%2Fdf10d%2Fimg%2Favatars%2Fava_0025-32.png", + "image_48": "https://secure.gravatar.com/avatar/071531b6d095f27a7cf6389dcfbb5766.jpg?s=48&d=https%3A%2F%2Fa.slack-edge.com%2Fdf10d%2Fimg%2Favatars%2Fava_0025-48.png", + "image_72": "https://secure.gravatar.com/avatar/071531b6d095f27a7cf6389dcfbb5766.jpg?s=72&d=https%3A%2F%2Fa.slack-edge.com%2Fdf10d%2Fimg%2Favatars%2Fava_0025-72.png", + "image_192": "https://secure.gravatar.com/avatar/071531b6d095f27a7cf6389dcfbb5766.jpg?s=192&d=https%3A%2F%2Fa.slack-edge.com%2Fdf10d%2Fimg%2Favatars%2Fava_0025-192.png", + "image_512": "https://secure.gravatar.com/avatar/071531b6d095f27a7cf6389dcfbb5766.jpg?s=512&d=https%3A%2F%2Fa.slack-edge.com%2Fdf10d%2Fimg%2Favatars%2Fava_0025-512.png", + "status_text_canonical": "", + "team": "T01CWUYP5AR" + }, + "is_admin": false, + "is_owner": false, + "is_primary_owner": false, + "is_restricted": false, + "is_ultra_restricted": false, + "is_bot": false, + "is_app_user": false, + "updated": 1694564822, + "is_email_confirmed": true, + "has_2fa": false, + "who_can_share_contact_card": "EVERYONE" + }, + "U05KKM07PJP": { + "id": "U05KKM07PJP", + "team_id": "T01CWUYP5AR", + "name": "peter.hicks", + "deleted": false, + "color": "dd8527", + "real_name": "Peter Hicks", + "tz": "America/Los_Angeles", + "tz_label": "Pacific Standard Time", + "tz_offset": -28800, + "profile": { + "title": "", + "phone": "", + "skype": "", + "real_name": "Peter Hicks", + "real_name_normalized": "Peter Hicks", + "display_name": "Peter Hicks", + "display_name_normalized": "Peter Hicks", + "fields": {}, + "status_text": "", + "status_emoji": "", + "status_emoji_display_info": [], + "status_expiration": 0, + "avatar_hash": "037cd5a411ed", + "image_original": "https://avatars.slack-edge.com/2023-07-31/5662890750117_037cd5a411ed6bbcf738_original.png", + "is_custom_image": true, + "email": "peter.hicks@astronomer.io", + "first_name": "Peter", + "last_name": "Hicks", + "image_24": "https://avatars.slack-edge.com/2023-07-31/5662890750117_037cd5a411ed6bbcf738_24.png", + "image_32": "https://avatars.slack-edge.com/2023-07-31/5662890750117_037cd5a411ed6bbcf738_32.png", + "image_48": "https://avatars.slack-edge.com/2023-07-31/5662890750117_037cd5a411ed6bbcf738_48.png", + "image_72": "https://avatars.slack-edge.com/2023-07-31/5662890750117_037cd5a411ed6bbcf738_72.png", + "image_192": "https://avatars.slack-edge.com/2023-07-31/5662890750117_037cd5a411ed6bbcf738_192.png", + "image_512": "https://avatars.slack-edge.com/2023-07-31/5662890750117_037cd5a411ed6bbcf738_512.png", + "image_1024": "https://avatars.slack-edge.com/2023-07-31/5662890750117_037cd5a411ed6bbcf738_1024.png", + "status_text_canonical": "", + "team": "T01CWUYP5AR" + }, + "is_admin": false, + "is_owner": false, + "is_primary_owner": false, + "is_restricted": false, + "is_ultra_restricted": false, + "is_bot": false, + "is_app_user": false, + "updated": 1695424773, + "is_email_confirmed": true, + "has_2fa": false, + "who_can_share_contact_card": "EVERYONE" + } +} \ No newline at end of file diff --git a/html/C01CK9T7HKR-0.html b/slack-archive/html/C01CK9T7HKR-0.html similarity index 77% rename from html/C01CK9T7HKR-0.html rename to slack-archive/html/C01CK9T7HKR-0.html index 722632b..778fe02 100644 --- a/html/C01CK9T7HKR-0.html +++ b/slack-archive/html/C01CK9T7HKR-0.html @@ -1,56 +1,10 @@ Slack

general

Created by Julien Le Dem on Tuesday, October 20th, 2020

OpenLineage spec discussion. See also: <https://github.com/OpenLineage/OpenLineage#community> -2023 Ecosystem Survey: <http://bit.ly/ecosystem_survey|bit.ly/ecosystem_survey>

ldaceySunday, August 20th, 2023 at 5:45:41 PM GMT-04:00
i saw OpenLineage was built into Airflow recently as a provider but the documentation seems really light (https://airflow.apache.org/docs/apache-airflow-providers-openlineage/stable/guides/user.html), is the documentation from openlineage the correct way I should proceed?

https://openlineage.io/docs/integrations/airflow/usage
👍1
Julien Le DemMonday, August 21st, 2023 at 8:26:56 PM GMT-04:00
openlineage-airflow is the package maintained in the OpenLineage project and to be used for versions of Airflow before 2.7. You could use it with 2.7 as well but you’d be staying on the “old” integration.
apache-airflow-providers-openlineage is the new package, maintained in the Airflow project that can be used starting Airflow 2.7 and is the recommended package moving forward. It is compatible with the configuration of the old package described in that usage page. CC: @Maciej Obuchowski @Jakub Dardziński It looks like this page needs improvement.
Maciej ObuchowskiTuesday, August 22nd, 2023 at 5:03:28 AM GMT-04:00
Yeah, I'll fix that
1
Jakub DardzińskiTuesday, August 22nd, 2023 at 5:55:08 PM GMT-04:00
🙌2
ldaceyTuesday, August 22nd, 2023 at 5:54:20 PM GMT-04:00
do I label certain raw data sources as a dataset, for example SFTP/FTP sites, 0365 emails, etc? I extract that data into a bucket for the client in a "folder" called "raw" which I know will be an OL Dataset. Would this GCS folder (after extracting the data with Airflow) be the first Dataset OL is aware of?

<gcs://client-bucket/source-system-lob/raw>

I then process that data into partitioned parquet datasets which would also be OL Datasets:
<gcs://client-bucket/source-system-lob/staging>
<gcs://client-bucket/source-system-lob/analytics>
Jakub DardzińskiTuesday, August 22nd, 2023 at 6:02:46 PM GMT-04:00
that really depends on the use case IMHO
if you consider a whole directory/folder as a dataset (meaning that each file inside folds into a larger whole) you should label dataset as directory

you might as well have directory with each file being something different - in this case it would be best to set each file separately as dataset
Jakub DardzińskiTuesday, August 22nd, 2023 at 6:04:32 PM GMT-04:00
there was also SymlinksDatasetFacet introduced to store alternative dataset names, might be useful: https://github.com/OpenLineage/OpenLineage/pull/936
ldaceyTuesday, August 22nd, 2023 at 6:07:26 PM GMT-04:00
cool, yeah in general each file is just a snapshot of data from a client (for example, daily dump). the parquet datasets are normally partitioned and might have small fragments and I definitely picture it as more of a table than individual files
👍1
Maciej ObuchowskiWednesday, August 23rd, 2023 at 8:22:09 AM GMT-04:00
Agree with Jakub here - with object storage, people use different patterns, but usually some directory layer vs file is the valid abstraction level, especially if your pattern is adding files with new data inside
👍1
ldaceyFriday, August 25th, 2023 at 10:26:52 AM GMT-04:00
I tested a dataset for each raw file versus the folder and the folder looks much cleaner (not sure if I can collapse individual datasets/files into a group?)

from 2022, this particular source had 6 raw schema changes (client controlled, no warning). what should I do to make that as obvious as possible if I track the dataset at a folder level?
ldaceyFriday, August 25th, 2023 at 10:32:19 AM GMT-04:00
I was thinking that I could name the dataset based on the schema_version (identified by the raw column names), so in this example I would have 6 OL datasets feeding into one "staging" dataset
ldaceyFriday, August 25th, 2023 at 10:32:57 AM GMT-04:00
not sure what the best practice would be in this scenario though
ldaceyTuesday, August 22nd, 2023 at 5:55:38 PM GMT-04:00
• also saw the docs reference URI = gs://{bucket name}{path} and I wondered if the path would include the filename, or if it was just the base path like I showed above
Mars LanTuesday, August 22nd, 2023 at 6:35:45 PM GMT-04:00
Has anyone managed to get the OL Airflow integration to work on AWS MWAA? We've tried pretty much every trick but still ended up with the following error:
Broken plugin: [openlineage.airflow.plugin] No module named 'openlineage.airflow'; 'openlineage' is not a package

Maciej ObuchowskiWednesday, August 23rd, 2023 at 5:22:18 AM GMT-04:00
Which version are you trying to use?
Maciej ObuchowskiWednesday, August 23rd, 2023 at 5:22:45 AM GMT-04:00
Both OL and MWAA/Airflow 🙂
Maciej ObuchowskiWednesday, August 23rd, 2023 at 5:23:52 AM GMT-04:00
'openlineage' is not a package

suggests that something went wrong with import process, for example cycle in import path
Mars LanWednesday, August 23rd, 2023 at 4:50:34 PM GMT-04:00
MWAA: 2.6.3
OL: 1.0.0

I can see from the log that OL has been successfully installed to the webserver:
Successfully installed openlineage-airflow-1.0.0 openlineage-integration-common-1.0.0 openlineage-python-1.0.0 openlineage-sql-1.0.0

This is the full stacktrace:
Traceback (most recent call last):
-
-File "/usr/local/airflow/.local/lib/python3.10/site-packages/airflow/plugins_manager.py", line 229, in load_entrypoint_plugins
-plugin_class = entry_point.load()
-File "/usr/local/airflow/.local/lib/python3.10/site-packages/importlib_metadata/__init__.py", line 209, in load
-module = import_module(match.group('module'))
-File "/usr/lib/python3.10/importlib/__init__.py", line 126, in import_module
-return _bootstrap._gcd_import(name[level:], package, level)
-File "&lt;frozen importlib._bootstrap&gt;", line 1050, in _gcd_import
-File "&lt;frozen importlib._bootstrap&gt;", line 1027, in _find_and_load
-File "&lt;frozen importlib._bootstrap&gt;", line 992, in _find_and_load_unlocked
-File "&lt;frozen importlib._bootstrap&gt;", line 241, in _call_with_frames_removed
-File "&lt;frozen importlib._bootstrap&gt;", line 1050, in _gcd_import
-File "&lt;frozen importlib._bootstrap&gt;", line 1027, in _find_and_load
-File "&lt;frozen importlib._bootstrap&gt;", line 1001, in _find_and_load_unlocked
-ModuleNotFoundError: No module named 'openlineage.airflow'; 'openlineage' is not a package
Jakub DardzińskiThursday, August 24th, 2023 at 8:18:36 AM GMT-04:00
It’s taking long to update MWAA environment but I tested 2.6.3 version with the followingrequirements.txt:
openlineage-airflow

and
openlineage-airflow==1.0.0

is there any step that might lead to some unexpected results?
Mars LanThursday, August 24th, 2023 at 8:29:30 AM GMT-04:00
Yeah, it takes forever to update MWAA even for a simple change. If you open either the webserver log (in CloudWatch) or the AirFlow UI, you should see the above error message.
Jakub DardzińskiThursday, August 24th, 2023 at 8:33:53 AM GMT-04:00
The thing is that I don’t see any error messages.
I wrote simple DAG to test too:
from __future__ import annotations
-
-from datetime import datetime
-
-from airflow.models import DAG
-
-try:
-    from airflow.operators.empty import EmptyOperator
-except ModuleNotFoundError:
-    from airflow.operators.dummy import DummyOperator as EmptyOperator  # type: ignore
-
-from openlineage.airflow.adapter import OpenLineageAdapter
-from openlineage.client.client import OpenLineageClient
-
-from airflow.operators.python import PythonOperator
-
-DAG_ID = "example_ol"
-
-def callable():
-    client = OpenLineageClient()
-    adapter = OpenLineageAdapter()
-    print(client, adapter)
-
-with DAG(
-    dag_id=DAG_ID,
-    start_date=datetime(2021, 1, 1),
-    schedule="@once",
-    catchup=False,
-) as dag:
-    begin = EmptyOperator(task_id="begin")
-
-    test = PythonOperator(task_id='print_client', python_callable=callable)

and it gives expected results as well
Mars LanThursday, August 24th, 2023 at 8:48:11 AM GMT-04:00
Oh how interesting. I did have a plugin that sets the endpoint & key via env var. Let me try to disable that to see if it fixes the issue. Will report back after 30 mins, or however long it takes to update MWAA 😉
Jakub DardzińskiThursday, August 24th, 2023 at 8:50:05 AM GMT-04:00
Mars LanThursday, August 24th, 2023 at 9:04:27 AM GMT-04:00
Actually no. I'm not aware of this guide. I assume it's outdated already?
Jakub DardzińskiThursday, August 24th, 2023 at 9:04:54 AM GMT-04:00
tbh I don’t know
Mars LanThursday, August 24th, 2023 at 9:04:55 AM GMT-04:00
Actually while we're on that topic, what's the recommended way to pass the URL & API Key in MWAA?
Jakub DardzińskiThursday, August 24th, 2023 at 9:28:00 AM GMT-04:00
I think it's still a plugin that sets env vars
Mars LanThursday, August 24th, 2023 at 9:32:18 AM GMT-04:00
Yeah based on the page you shared, secret manager + plugin seems like the way to go.
Mars LanThursday, August 24th, 2023 at 10:31:50 AM GMT-04:00
Alas after disabling the plugin and restarting the cluster, I'm still getting the same error. Do you mind to share a screenshot of your cluster's settings so I can compare?
Maciej ObuchowskiThursday, August 24th, 2023 at 11:57:04 AM GMT-04:00
Are you maybe importing some top level OpenLineage code anywhere? This error is most likely circular import
Mars LanThursday, August 24th, 2023 at 12:01:12 PM GMT-04:00
Let me try removing all the dags to see if it helps.
Mars LanThursday, August 24th, 2023 at 6:42:49 PM GMT-04:00
@Maciej Obuchowski you were correct! It was indeed the DAGs. The errors are gone after removing all the dags. Now just need to figure what caused the circular import since I didn't import OL directly in DAG.
Mars LanThursday, August 24th, 2023 at 6:44:33 PM GMT-04:00
Could this be the issue?
from airflow.lineage.entities import File, Table

How could I declare lineage manually if I can't import these classes?
Maciej ObuchowskiFriday, August 25th, 2023 at 6:52:47 AM GMT-04:00
@Mars Lan I'll look in more details next week, as I'm in transit now
Maciej ObuchowskiFriday, August 25th, 2023 at 6:53:18 AM GMT-04:00
but if you could narrow down a problem to single dag that I or @Jakub Dardziński could reproduce, ideally locally, it would help a lot
Mars LanFriday, August 25th, 2023 at 7:07:11 AM GMT-04:00
Thanks. I think I understand how this works much better now. Found a few useful BQ example dags. Will give them a try and report back.
🔥2
NitinWednesday, August 23rd, 2023 at 7:14:44 AM GMT-04:00
Hi All,
I want to capture, source and target table details as lineage information with openlineage for Amazon Redshift. Please let me know, if anyone has done it
Jakub DardzińskiWednesday, August 23rd, 2023 at 7:32:19 AM GMT-04:00
are you using Airflow to connect to Redshift?
NitinThursday, August 24th, 2023 at 6:50:05 AM GMT-04:00
Hi @Jakub Dardziński,
Thank you for your reply.
No, we are not using Airflow.
We are using load/Unload cmd with Pyspark and also Pandas with JDBC connection
Jakub DardzińskiFriday, August 25th, 2023 at 1:28:37 PM GMT-04:00
@Paweł Leszczyński might know answer if Spark<->OL integration works with Redshift. Eventually JDBC is supported with sqlparser

for Pandas I think there wasn’t too much work done
Paweł LeszczyńskiMonday, August 28th, 2023 at 2:18:49 AM GMT-04:00
@Nitin If you're using jdbc within Spark, the lineage should be obtained via sqlparser-rs library https://github.com/sqlparser-rs/sqlparser-rs. In case it's not, please try to provide some minimal SQL code (or pyspark) which leads to uncaught lineage.
NitinMonday, August 28th, 2023 at 4:53:03 AM GMT-04:00
Hi @Jakub Dardziński / @Paweł Leszczyński, thank you for taking out time to reply on my query. We need to capture only load and unload query lineage which we are running using Spark.

If you have any sample implementation for reference, it will be indeed helpful
Paweł LeszczyńskiMonday, August 28th, 2023 at 6:12:46 AM GMT-04:00
NitinMonday, August 28th, 2023 at 8:18:14 AM GMT-04:00
Yeah! any way you can think of, we can accommodate it specially load and unload statement.
Also, we would like to capture, lineage information where our endpoints are Sagemaker and Redis
NitinMonday, August 28th, 2023 at 1:20:37 PM GMT-04:00
@Paweł Leszczyński can we use this code base integration/common/openlineage/common/provider/redshift_data.py for redshift lineage capture
Jakub DardzińskiMonday, August 28th, 2023 at 2:26:40 PM GMT-04:00
it still expects input and output tables that are usually retrieved from sqlparser
Jakub DardzińskiMonday, August 28th, 2023 at 2:31:00 PM GMT-04:00
AbdallahWednesday, August 23rd, 2023 at 10:55:10 AM GMT-04:00
Approve a new release please 🙂
• Fix spark integration filtering Databricks events.
11
Michael RobinsonWednesday, August 23rd, 2023 at 12:27:15 PM GMT-04:00
Thank you for requesting a release @Abdallah. Three +1s from committers will authorize.
🙌1
Michael RobinsonWednesday, August 23rd, 2023 at 1:13:18 PM GMT-04:00
Thanks, all. The release is authorized and will be initiated within 2 business days.
Athitya KumarWednesday, August 23rd, 2023 at 1:08:48 PM GMT-04:00
Hey folks! Do we have clear step-by-step documentation on how we can leverage the ServiceLoader based approach for injecting specific OpenLineage customisations for tweaking the transport type with defaults / tweaking column level lineage etc?
Maciej ObuchowskiWednesday, August 23rd, 2023 at 1:29:05 PM GMT-04:00
For custom transport, you have to provide implementation of interface https://github.com/OpenLineage/OpenLineage/blob/4a1a5c3bf9767467b71ca0e1b6d820ba9e[…]ain/java/io/openlineage/client/transports/TransportBuilder.java and point to it in META_INF file
Maciej ObuchowskiWednesday, August 23rd, 2023 at 1:29:52 PM GMT-04:00
But if I understand correctly, if you want to change behavior rather than extend, the correct way may be to either contribute it to repo - if that behavior is useful to anyone, or fork the repo
Athitya KumarWednesday, August 23rd, 2023 at 3:14:43 PM GMT-04:00
@Maciej Obuchowski - Can you elaborate more on the "point to it in META_INF file"? Let's say we have the custom transport type built in a standalone jar by extending transport builder - what're the exact next steps to use this custom transport in the standalone jar when doing spark-submit?
Maciej ObuchowskiWednesday, August 23rd, 2023 at 3:23:13 PM GMT-04:00
@Athitya Kumar your jar needs to have META-INF/services/io.openlineage.client.transports.TransportBuilder with fully qualified class names of your custom TransportBuilders there - like openlineage-spark has
io.openlineage.client.transports.HttpTransportBuilder
+2023 Ecosystem Survey: <http://bit.ly/ecosystem_survey|bit.ly/ecosystem_survey>

AbdallahWednesday, August 23rd, 2023 at 10:55:10 AM GMT-04:00
Approve a new release please 🙂
• Fix spark integration filtering Databricks events.
11
Michael RobinsonWednesday, August 23rd, 2023 at 12:27:15 PM GMT-04:00
Thank you for requesting a release @Abdallah. Three +1s from committers will authorize.
🙌1
Michael RobinsonWednesday, August 23rd, 2023 at 1:13:18 PM GMT-04:00
Thanks, all. The release is authorized and will be initiated within 2 business days.
Athitya KumarWednesday, August 23rd, 2023 at 1:08:48 PM GMT-04:00
Hey folks! Do we have clear step-by-step documentation on how we can leverage the ServiceLoader based approach for injecting specific OpenLineage customisations for tweaking the transport type with defaults / tweaking column level lineage etc?
Maciej ObuchowskiWednesday, August 23rd, 2023 at 1:29:05 PM GMT-04:00
For custom transport, you have to provide implementation of interface https://github.com/OpenLineage/OpenLineage/blob/4a1a5c3bf9767467b71ca0e1b6d820ba9e[…]ain/java/io/openlineage/client/transports/TransportBuilder.java and point to it in META_INF file
Maciej ObuchowskiWednesday, August 23rd, 2023 at 1:29:52 PM GMT-04:00
But if I understand correctly, if you want to change behavior rather than extend, the correct way may be to either contribute it to repo - if that behavior is useful to anyone, or fork the repo
Athitya KumarWednesday, August 23rd, 2023 at 3:14:43 PM GMT-04:00
@Maciej Obuchowski - Can you elaborate more on the "point to it in META_INF file"? Let's say we have the custom transport type built in a standalone jar by extending transport builder - what're the exact next steps to use this custom transport in the standalone jar when doing spark-submit?
Maciej ObuchowskiWednesday, August 23rd, 2023 at 3:23:13 PM GMT-04:00
@Athitya Kumar your jar needs to have META-INF/services/io.openlineage.client.transports.TransportBuilder with fully qualified class names of your custom TransportBuilders there - like openlineage-spark has
io.openlineage.client.transports.HttpTransportBuilder
 io.openlineage.client.transports.KafkaTransportBuilder
 io.openlineage.client.transports.ConsoleTransportBuilder
 io.openlineage.client.transports.FileTransportBuilder
-io.openlineage.client.transports.KinesisTransportBuilder
Athitya KumarFriday, August 25th, 2023 at 1:49:29 AM GMT-04:00
@Maciej Obuchowski - I think this change may be required for consumers to leverage custom transports, can you check & verify this GH comment?
https://github.com/OpenLineage/OpenLineage/issues/2007#issuecomment-1690350630
Maciej ObuchowskiFriday, August 25th, 2023 at 6:52:30 AM GMT-04:00
Probably, I will look at more details next week @Athitya Kumar as I'm in transit
👍1
Michael RobinsonWednesday, August 23rd, 2023 at 3:04:10 PM GMT-04:00
@channel
We released OpenLineage 1.1.0, including:
Additions:
• Flink: create Openlineage configuration based on Flink configuration #2033 @pawel-big-lebowski
• Java: add Javadocs to the Java client #2004 @julienledem
• Spark: append output dataset name to a job name #2036 @pawel-big-lebowski
• Spark: support Spark 3.4.1 #2057 @pawel-big-lebowski
Fixes:
• Flink: fix a bug when getting schema for KafkaSink #2042 @pentium3
• Spark: fix ignored event adaptive_spark_plan in Databricks #2061 @algorithmy1
Plus additional bug fixes, doc changes and more.
Thanks to all the contributors, especially new contributors @pentium3 and @Abdallah!
Release: https://github.com/OpenLineage/OpenLineage/releases/tag/1.1.0
Changelog: https://github.com/OpenLineage/OpenLineage/blob/main/CHANGELOG.md
Commit history: https://github.com/OpenLineage/OpenLineage/compare/1.0.0...1.1.0
Maven: https://oss.sonatype.org/#nexus-search;quick~openlineage
PyPI: https://pypi.org/project/openlineage-python/
👏9
1
Michael RobinsonFriday, August 25th, 2023 at 10:29:23 AM GMT-04:00
@channel
Friendly reminder: our next in-person meetup is next Wednesday, August 30th in San Francisco at Astronomer’s offices in the Financial District. You can sign up and find the details on the meetup event page.
George PolychronopoulosFriday, August 25th, 2023 at 10:57:30 AM GMT-04:00
hi Openlineage team , we would like to join one of your meetups(me and @Madhav Kakumani nad @Phil Rolph and we're wondering if you are hosting any meetups after the 18/9 ? We are trying to join this but air - tickets are quite expensive
Harel SheinFriday, August 25th, 2023 at 11:32:12 AM GMT-04:00
there will certainly be more meetups, don’t worry about that!
Harel SheinFriday, August 25th, 2023 at 11:32:30 AM GMT-04:00
where are you located? perhaps we can try to organize a meetup closer to where you are.
George PolychronopoulosFriday, August 25th, 2023 at 11:49:37 AM GMT-04:00
Thanks a lot for the response, we are in London. We'd be glad to help you organise a meetup and also meet in person!
Michael RobinsonFriday, August 25th, 2023 at 11:51:39 AM GMT-04:00
This is awesome, thanks @George Polychronopoulos. I’ll start a channel and invite you
Juan Luis Cano RodríguezMonday, August 28th, 2023 at 4:47:53 AM GMT-04:00
hi folks, I'm looking into exporting static metadata, and found that DatasetEvent requires a eventTime, which in my mind doesn't make sense for static events. I'm setting it to None and the Python client seems to work, but wanted to ask if I'm missing something.
Paweł LeszczyńskiMonday, August 28th, 2023 at 5:59:10 AM GMT-04:00
Although you emit DatasetEvent, you still emit an event and eventTime is a valid marker.
Juan Luis Cano RodríguezMonday, August 28th, 2023 at 6:01:40 AM GMT-04:00
so, should I use the current time at the moment of emitting it and that's it?
Paweł LeszczyńskiMonday, August 28th, 2023 at 6:01:53 AM GMT-04:00
yes, that should be it
1
Juan Luis Cano RodríguezMonday, August 28th, 2023 at 4:49:21 AM GMT-04:00
and something else: I understand that Marquez does not yet support the 2.0 spec, hence it's incompatible with static metadata right? I tried to emit a list of DatasetEvent s and got HTTPError: 422 Client Error: Unprocessable Entity for url: <http://localhost:3000/api/v1/lineage> (I'm using a FileTransport for now)
Paweł LeszczyńskiMonday, August 28th, 2023 at 6:02:49 AM GMT-04:00
marquez is not capable of reflecting DatasetEvents in DB but it should respond with Unsupported event type
Paweł LeszczyńskiMonday, August 28th, 2023 at 6:03:15 AM GMT-04:00
and return 200 instead of 201 created
Juan Luis Cano RodríguezMonday, August 28th, 2023 at 6:05:41 AM GMT-04:00
I'll have a deeper look then, probably I'm doing something wrong. thanks @Paweł Leszczyński
Joshua DotsonMonday, August 28th, 2023 at 1:25:58 PM GMT-04:00
Hi folks. I have some pure golang jobs from which I need to emit OL events to Marquez. Is the right way to go about this to generate a Golang client from the Marquez OpenAPI spec and use that client from my go jobs?
Jakub DardzińskiMonday, August 28th, 2023 at 2:23:24 PM GMT-04:00
I'd rather generate them from OL spec (compliant with JSON Schema)
Joshua DotsonMonday, August 28th, 2023 at 3:12:21 PM GMT-04:00
I'll look into this. I take you to mean that I would use the OL spec which is available as a set of JSON schemas to create the data object and then HTTP POST it using vanilla Golang. Is that correct? Thank you for your help!
Jakub DardzińskiMonday, August 28th, 2023 at 3:30:05 PM GMT-04:00
Correct! You’re also very welcome to contribute Golang client (currently we have Python & Java clients) if you manage to send events using golang 🙂
👏1
Michael RobinsonMonday, August 28th, 2023 at 5:28:31 PM GMT-04:00
@channel
The agenda for the Toronto Meetup at Airflow Summit on 9/18 has been updated. This promises to be an exciting, richly productive discussion. Don’t miss it if you’ll be in the area!
1. Intros
2. Evolution of spec presentation/discussion (project background/history)
3. State of the community
4. Spark/Column lineage update
5. Airflow Provider update
6. Roadmap Discussion
7. Action items review/next steps
❤️3
Michael RobinsonMonday, August 28th, 2023 at 8:05:37 PM GMT-04:00
New on the OpenLineage blog: a close look at the new OpenLineage Airflow Provider, including:
• the critical improvements it brings to the integration
• the high-level design
• implementation details
• an example operator
• planned enhancements
• a list of supported operators
• more.
The post, by @Maciej Obuchowski, @Julien Le Dem and myself is live now on the OpenLineage blog.
🎉5
Sarwat FatimaTuesday, August 29th, 2023 at 3:18:04 AM GMT-04:00
Hello, I'm currently in the process of following the instructions outlined in the provided getting started guide at https://openlineage.io/getting-started/. However, I've encountered a problem while attempting to complete Step 1 of the guide. Unfortunately, I'm encountering an internal server error at this stage. I did manage to successfully run Marquez, but it appears that there might be an issue that needs to be addressed. I have attached screen shots.
Jakub DardzińskiTuesday, August 29th, 2023 at 3:20:18 AM GMT-04:00
is 5000 port taken by any other application? or ./docker/up.sh has some errors in logs?
Sarwat FatimaTuesday, August 29th, 2023 at 5:23:01 AM GMT-04:00
@Jakub Dardziński 5000 port is not taken by any other application. The logs show some errors but I am not sure what is the issue here.
Maciej ObuchowskiTuesday, August 29th, 2023 at 10:02:38 AM GMT-04:00
I think Marquez is running on WSL while you're trying to connect from host computer?
Juan Luis Cano RodríguezTuesday, August 29th, 2023 at 5:20:39 AM GMT-04:00
hi folks, for now I'm producing .jsonl (or .ndjson ) files with one event per line, do you know if there's any way to validate those? would standard JSON Schema tools work?
Juan Luis Cano RodríguezTuesday, August 29th, 2023 at 10:58:29 AM GMT-04:00
reply by @U0544QC1DS9: yes 🙂💯
👍1
ldaceyTuesday, August 29th, 2023 at 1:12:32 PM GMT-04:00
for namespaces, if my data is moving between sources (SFTP -> GCS -> Azure Blob (synapse connects to parquet datasets) then should my namespace be based on the client I am working with? my current namespace has been to refer to the bucket, but that falls apart when considering the data sources and some destinations. perhaps I should just add a field for client-name instead to have a consolidated view?
Maciej ObuchowskiWednesday, August 30th, 2023 at 10:53:08 AM GMT-04:00
> then should my namespace be based on the client I am working with?
I think each of those sources should be a different namespace?
ldaceyWednesday, August 30th, 2023 at 12:59:53 PM GMT-04:00
got it, yeah I was kind of picturing as one namespace for the client (we handle many clients but they are completely distinct entities). I was able to get it to work with multiple namespaces like you suggested and Marquez was able to plot everything correctly in the visualization
ldaceyWednesday, August 30th, 2023 at 1:01:18 PM GMT-04:00
I noticed some of my Dataset facets make more sense as Run facets, for example, the name of the specific file I processed and how many rows of data / size of the data for that schedule. that won't impact the Run facets Airflow provides right? I can still have the schedule information + my custom run facets?
Maciej ObuchowskiWednesday, August 30th, 2023 at 1:06:38 PM GMT-04:00
Yes, unless you name it the same as one of the Airflow facets 🙂
GitHubOpenLineageIssuesWednesday, August 30th, 2023 at 8:15:29 AM GMT-04:00
Hi, Will really appreciate if someone can guide me or provide me any pointer - if they have been able to implement authentication/authorization for access to Marquez. Have not seen much info around it. Any pointers greatly appreciated. Thanks in advance.
Julien Le DemWednesday, August 30th, 2023 at 12:23:18 PM GMT-04:00
I’ve seen people do this through the ingress controller in Kubernetes. Unfortunately I don’t have documentation besides k8s specific ones you would find for the ingress controller you’re using. You’d redirect any unauthenticated request to your identity provider
1
Michael RobinsonWednesday, August 30th, 2023 at 11:50:05 AM GMT-04:00
@channel
Friendly reminder: there’s a meetup tonight at Astronomer’s offices in SF!
1
Julien Le DemWednesday, August 30th, 2023 at 12:15:31 PM GMT-04:00
I’ll be there and looking forward to see @John Lukenoff ‘s presentation
Michael BarrientosWednesday, August 30th, 2023 at 9:38:31 PM GMT-04:00
Can anyone let 3 people stuck downstairs into the 7th floor?
👍1
Willy LulciucWednesday, August 30th, 2023 at 11:25:21 PM GMT-04:00
Sorry about that!
YunheThursday, August 31st, 2023 at 2:31:48 AM GMT-04:00
hello,everyone,i can run openLineage spark code in my notebook with python,but when use my idea to execute scala code like this:
import org.apache.spark.internal.Logging
import org.apache.spark.sql.SparkSession
import io.openlineage.client.OpenLineageClientUtils.loadOpenLineageYaml
import org.apache.spark.scheduler.{SparkListener, SparkListenerApplicationEnd, SparkListenerApplicationStart}
import sun.java2d.marlin.MarlinUtils.logInfo
object Test {
def main(args: Array[String]): Unit = {

val spark = SparkSession
.builder()
.master("local")
.appName("test")
.config("spark.jars.packages","io.openlineage:openlineage-spark:0.12.0")
.config("spark.extraListeners","io.openlineage.spark.agent.OpenLineageSparkListener")
.config("spark.openlineage.transport.type","console")
.getOrCreate()

spark.sparkContext.setLogLevel("INFO")

//spark.sparkContext.addSparkListener(new MySparkAppListener)
import spark.implicits._
val input = Seq((1, "zs", 2020), (2, "ls", 2023)).toDF("id", "name", "year")

input.select("id", "name").orderBy("id").show()

}

}

there is something wrong:
Exception in thread "spark-listener-group-shared" java.lang.NoSuchMethodError: io.openlineage.client.OpenLineageClientUtils.loadOpenLineageYaml(Ljava/io/InputStream;)Lio/openlineage/client/OpenLineageYaml;
at io.openlineage.spark.agent.ArgumentParser.extractOpenlineageConfFromSparkConf(ArgumentParser.java:114)
at io.openlineage.spark.agent.ArgumentParser.parse(ArgumentParser.java:78)
at io.openlineage.spark.agent.OpenLineageSparkListener.initializeContextFactoryIfNotInitialized(OpenLineageSparkListener.java:277)
at io.openlineage.spark.agent.OpenLineageSparkListener.onApplicationStart(OpenLineageSparkListener.java:267)
at org.apache.spark.scheduler.SparkListenerBus.doPostEvent(SparkListenerBus.scala:55)
at org.apache.spark.scheduler.SparkListenerBus.doPostEvent$(SparkListenerBus.scala:28)
at org.apache.spark.scheduler.AsyncEventQueue.doPostEvent(AsyncEventQueue.scala:37)
at org.apache.spark.scheduler.AsyncEventQueue.doPostEvent(AsyncEventQueue.scala:37)
at org.apache.spark.util.ListenerBus.postToAll(ListenerBus.scala:117)
at org.apache.spark.util.ListenerBus.postToAll$(ListenerBus.scala:101)
at org.apache.spark.scheduler.AsyncEventQueue.super$postToAll(AsyncEventQueue.scala:105)
at org.apache.spark.scheduler.AsyncEventQueue.$anonfun$dispatch$1(AsyncEventQueue.scala:105)
at scala.runtime.java8.JFunction0$mcJ$sp.apply(JFunction0$mcJ$sp.java:23)
at scala.util.DynamicVariable.withValue(DynamicVariable.scala:62)
at org.apache.spark.scheduler.AsyncEventQueue.org$apache$spark$scheduler$AsyncEventQueue$$dispatch(AsyncEventQueue.scala:100)
at org.apache.spark.scheduler.AsyncEventQueue$$anon$2.$anonfun$run$1(AsyncEventQueue.scala:96)
at org.apache.spark.util.Utils$.tryOrStopSparkContext(Utils.scala:1446)
at org.apache.spark.scheduler.AsyncEventQueue$$anon$2.run(AsyncEventQueue.scala:96)

i want to know how can i set idea scala environment correctly
Paweł LeszczyńskiThursday, August 31st, 2023 at 2:58:41 AM GMT-04:00
io.openlineage:openlineage-spark:0.12.0 -> could you repeat the steps with newer version?
YunheThursday, August 31st, 2023 at 3:51:52 AM GMT-04:00
ok,it`s my first use thie lineage tool. first,I added dependences in my pom.xml like this:
<dependency>
<groupId>io.openlineage</groupId>
<artifactId>openlineage-java</artifactId>
<version>0.12.0</version>
</dependency>
<dependency>
<groupId>org.apache.logging.log4j</groupId>
<artifactId>log4j-api</artifactId>
<version>2.7</version>
</dependency>
<dependency>
<groupId>org.apache.logging.log4j</groupId>
<artifactId>log4j-core</artifactId>
<version>2.7</version>
</dependency>
<dependency>
<groupId>org.apache.logging.log4j</groupId>
<artifactId>log4j-slf4j-impl</artifactId>
<version>2.7</version>
</dependency>
<dependency>
<groupId>io.openlineage</groupId>
<artifactId>openlineage-spark</artifactId>
<version>0.30.1</version>
</dependency>

my spark version is 3.3.1 and the version can not change

second, in file Openlineage/intergration/spark I enter command : docker-compose up and follow the steps in this doc:
https://openlineage.io/docs/integrations/spark/quickstart_local
there is no erro when i use notebook to execute pyspark for openlineage and I could get json message.
but after I enter "docker-compose up" ,I want to use my Idea tool to execute scala code like above,the erro happend like above. It seems that I does not configure the environment correctly. so how can i fix the problem .
Paweł LeszczyńskiFriday, September 1st, 2023 at 5:15:28 AM GMT-04:00
please use latest io.openlineage:openlineage-spark:1.1.0 instead. openlineage-java is already contained in the jar, no need to add it on your own.
Sheeri Cabral (Collibra)Thursday, August 31st, 2023 at 3:33:19 PM GMT-04:00
Will the August meeting be put up at https://wiki.lfaidata.foundation/display/OpenLineage/Monthly+TSC+meeting soon? (usually it’s up in a few days 🙂
Maciej ObuchowskiFriday, September 1st, 2023 at 6:00:53 AM GMT-04:00
@Michael Robinson
Michael RobinsonFriday, September 1st, 2023 at 5:13:32 PM GMT-04:00
The recording is on the youtube channel here. I’ll update the wiki ASAP
1
Julien Le DemThursday, August 31st, 2023 at 6:10:20 PM GMT-04:00
🎉13
🙌10
❤️5
Julien Le DemFriday, September 1st, 2023 at 11:09:55 PM GMT-04:00
Michael RobinsonFriday, September 1st, 2023 at 5:16:21 PM GMT-04:00
@channel
The latest issue of OpenLineage News is out now! Please subscribe to get it directly in your inbox each month.
🙌2
1
Anirudh ShrinivasonMonday, September 4th, 2023 at 3:38:28 AM GMT-04:00
Hi guys, I'd like to capture the spark.databricks.clusterUsageTags.clusterAllTags property from databricks. However, the value of this is a list of keys, and therefore cannot be supported by custom environment facet builder.
I was thinking that capturing this property might be useful for most databricks workloads, and whether it might make sense to auto-capture it along with other databricks variables, similar to how we capture mount points for the databricks jobs.
Does this sound okay? If so, then I can help to contribute this functionality
Maciej ObuchowskiMonday, September 4th, 2023 at 6:43:47 AM GMT-04:00
Sounds good to me
Anirudh ShrinivasonMonday, September 11th, 2023 at 5:15:03 AM GMT-04:00
Anirudh ShrinivasonMonday, September 4th, 2023 at 6:39:05 AM GMT-04:00
Also, another small clarification is that when using MergeIntoCommand, I'm receiving the lineage events on the backend, but I cannot seem to find any logging of the payload when I enable debug mode in openlineage. I remember there was a similar issue reported by another user in the past. May I check if it might be possible to help with this? It's making debugging quite hard for these cases. Thanks!
Maciej ObuchowskiMonday, September 4th, 2023 at 6:54:12 AM GMT-04:00
I think it only depends on log4j configuration
Maciej ObuchowskiMonday, September 4th, 2023 at 6:57:15 AM GMT-04:00
# Set everything to be logged to the console
+io.openlineage.client.transports.KinesisTransportBuilder
Athitya KumarFriday, August 25th, 2023 at 1:49:29 AM GMT-04:00
@Maciej Obuchowski - I think this change may be required for consumers to leverage custom transports, can you check & verify this GH comment?
https://github.com/OpenLineage/OpenLineage/issues/2007#issuecomment-1690350630
Maciej ObuchowskiFriday, August 25th, 2023 at 6:52:30 AM GMT-04:00
Probably, I will look at more details next week @Athitya Kumar as I'm in transit
👍1
Michael RobinsonWednesday, August 23rd, 2023 at 3:04:10 PM GMT-04:00
@channel
We released OpenLineage 1.1.0, including:
Additions:
• Flink: create Openlineage configuration based on Flink configuration #2033 @pawel-big-lebowski
• Java: add Javadocs to the Java client #2004 @julienledem
• Spark: append output dataset name to a job name #2036 @pawel-big-lebowski
• Spark: support Spark 3.4.1 #2057 @pawel-big-lebowski
Fixes:
• Flink: fix a bug when getting schema for KafkaSink #2042 @pentium3
• Spark: fix ignored event adaptive_spark_plan in Databricks #2061 @algorithmy1
Plus additional bug fixes, doc changes and more.
Thanks to all the contributors, especially new contributors @pentium3 and @Abdallah!
Release: https://github.com/OpenLineage/OpenLineage/releases/tag/1.1.0
Changelog: https://github.com/OpenLineage/OpenLineage/blob/main/CHANGELOG.md
Commit history: https://github.com/OpenLineage/OpenLineage/compare/1.0.0...1.1.0
Maven: https://oss.sonatype.org/#nexus-search;quick~openlineage
PyPI: https://pypi.org/project/openlineage-python/
👏9
1
Michael RobinsonFriday, August 25th, 2023 at 10:29:23 AM GMT-04:00
@channel
Friendly reminder: our next in-person meetup is next Wednesday, August 30th in San Francisco at Astronomer’s offices in the Financial District. You can sign up and find the details on the meetup event page.
George PolychronopoulosFriday, August 25th, 2023 at 10:57:30 AM GMT-04:00
hi Openlineage team , we would like to join one of your meetups(me and @Madhav Kakumani nad @Phil Rolph and we're wondering if you are hosting any meetups after the 18/9 ? We are trying to join this but air - tickets are quite expensive
Harel SheinFriday, August 25th, 2023 at 11:32:12 AM GMT-04:00
there will certainly be more meetups, don’t worry about that!
Harel SheinFriday, August 25th, 2023 at 11:32:30 AM GMT-04:00
where are you located? perhaps we can try to organize a meetup closer to where you are.
George PolychronopoulosFriday, August 25th, 2023 at 11:49:37 AM GMT-04:00
Thanks a lot for the response, we are in London. We'd be glad to help you organise a meetup and also meet in person!
Michael RobinsonFriday, August 25th, 2023 at 11:51:39 AM GMT-04:00
This is awesome, thanks @George Polychronopoulos. I’ll start a channel and invite you
Juan Luis Cano RodríguezMonday, August 28th, 2023 at 4:47:53 AM GMT-04:00
hi folks, I'm looking into exporting static metadata, and found that DatasetEvent requires a eventTime, which in my mind doesn't make sense for static events. I'm setting it to None and the Python client seems to work, but wanted to ask if I'm missing something.
Paweł LeszczyńskiMonday, August 28th, 2023 at 5:59:10 AM GMT-04:00
Although you emit DatasetEvent, you still emit an event and eventTime is a valid marker.
Juan Luis Cano RodríguezMonday, August 28th, 2023 at 6:01:40 AM GMT-04:00
so, should I use the current time at the moment of emitting it and that's it?
Paweł LeszczyńskiMonday, August 28th, 2023 at 6:01:53 AM GMT-04:00
yes, that should be it
1
Juan Luis Cano RodríguezMonday, August 28th, 2023 at 4:49:21 AM GMT-04:00
and something else: I understand that Marquez does not yet support the 2.0 spec, hence it's incompatible with static metadata right? I tried to emit a list of DatasetEvent s and got HTTPError: 422 Client Error: Unprocessable Entity for url: <http://localhost:3000/api/v1/lineage> (I'm using a FileTransport for now)
Paweł LeszczyńskiMonday, August 28th, 2023 at 6:02:49 AM GMT-04:00
marquez is not capable of reflecting DatasetEvents in DB but it should respond with Unsupported event type
Paweł LeszczyńskiMonday, August 28th, 2023 at 6:03:15 AM GMT-04:00
and return 200 instead of 201 created
Juan Luis Cano RodríguezMonday, August 28th, 2023 at 6:05:41 AM GMT-04:00
I'll have a deeper look then, probably I'm doing something wrong. thanks @Paweł Leszczyński
Joshua DotsonMonday, August 28th, 2023 at 1:25:58 PM GMT-04:00
Hi folks. I have some pure golang jobs from which I need to emit OL events to Marquez. Is the right way to go about this to generate a Golang client from the Marquez OpenAPI spec and use that client from my go jobs?
Jakub DardzińskiMonday, August 28th, 2023 at 2:23:24 PM GMT-04:00
I'd rather generate them from OL spec (compliant with JSON Schema)
Joshua DotsonMonday, August 28th, 2023 at 3:12:21 PM GMT-04:00
I'll look into this. I take you to mean that I would use the OL spec which is available as a set of JSON schemas to create the data object and then HTTP POST it using vanilla Golang. Is that correct? Thank you for your help!
Jakub DardzińskiMonday, August 28th, 2023 at 3:30:05 PM GMT-04:00
Correct! You’re also very welcome to contribute Golang client (currently we have Python & Java clients) if you manage to send events using golang 🙂
👏1
Michael RobinsonMonday, August 28th, 2023 at 5:28:31 PM GMT-04:00
@channel
The agenda for the Toronto Meetup at Airflow Summit on 9/18 has been updated. This promises to be an exciting, richly productive discussion. Don’t miss it if you’ll be in the area!
1. Intros
2. Evolution of spec presentation/discussion (project background/history)
3. State of the community
4. Spark/Column lineage update
5. Airflow Provider update
6. Roadmap Discussion
7. Action items review/next steps
❤️3
Michael RobinsonMonday, August 28th, 2023 at 8:05:37 PM GMT-04:00
New on the OpenLineage blog: a close look at the new OpenLineage Airflow Provider, including:
• the critical improvements it brings to the integration
• the high-level design
• implementation details
• an example operator
• planned enhancements
• a list of supported operators
• more.
The post, by @Maciej Obuchowski, @Julien Le Dem and myself is live now on the OpenLineage blog.
🎉5
Sarwat FatimaTuesday, August 29th, 2023 at 3:18:04 AM GMT-04:00
Hello, I'm currently in the process of following the instructions outlined in the provided getting started guide at https://openlineage.io/getting-started/. However, I've encountered a problem while attempting to complete Step 1 of the guide. Unfortunately, I'm encountering an internal server error at this stage. I did manage to successfully run Marquez, but it appears that there might be an issue that needs to be addressed. I have attached screen shots.
Jakub DardzińskiTuesday, August 29th, 2023 at 3:20:18 AM GMT-04:00
is 5000 port taken by any other application? or ./docker/up.sh has some errors in logs?
Sarwat FatimaTuesday, August 29th, 2023 at 5:23:01 AM GMT-04:00
@Jakub Dardziński 5000 port is not taken by any other application. The logs show some errors but I am not sure what is the issue here.
Maciej ObuchowskiTuesday, August 29th, 2023 at 10:02:38 AM GMT-04:00
I think Marquez is running on WSL while you're trying to connect from host computer?
Juan Luis Cano RodríguezTuesday, August 29th, 2023 at 5:20:39 AM GMT-04:00
hi folks, for now I'm producing .jsonl (or .ndjson ) files with one event per line, do you know if there's any way to validate those? would standard JSON Schema tools work?
Juan Luis Cano RodríguezTuesday, August 29th, 2023 at 10:58:29 AM GMT-04:00
reply by @U0544QC1DS9: yes 🙂💯
👍1
ldaceyTuesday, August 29th, 2023 at 1:12:32 PM GMT-04:00
for namespaces, if my data is moving between sources (SFTP -> GCS -> Azure Blob (synapse connects to parquet datasets) then should my namespace be based on the client I am working with? my current namespace has been to refer to the bucket, but that falls apart when considering the data sources and some destinations. perhaps I should just add a field for client-name instead to have a consolidated view?
Maciej ObuchowskiWednesday, August 30th, 2023 at 10:53:08 AM GMT-04:00
> then should my namespace be based on the client I am working with?
I think each of those sources should be a different namespace?
ldaceyWednesday, August 30th, 2023 at 12:59:53 PM GMT-04:00
got it, yeah I was kind of picturing as one namespace for the client (we handle many clients but they are completely distinct entities). I was able to get it to work with multiple namespaces like you suggested and Marquez was able to plot everything correctly in the visualization
ldaceyWednesday, August 30th, 2023 at 1:01:18 PM GMT-04:00
I noticed some of my Dataset facets make more sense as Run facets, for example, the name of the specific file I processed and how many rows of data / size of the data for that schedule. that won't impact the Run facets Airflow provides right? I can still have the schedule information + my custom run facets?
Maciej ObuchowskiWednesday, August 30th, 2023 at 1:06:38 PM GMT-04:00
Yes, unless you name it the same as one of the Airflow facets 🙂
GitHubOpenLineageIssuesWednesday, August 30th, 2023 at 8:15:29 AM GMT-04:00
Hi, Will really appreciate if someone can guide me or provide me any pointer - if they have been able to implement authentication/authorization for access to Marquez. Have not seen much info around it. Any pointers greatly appreciated. Thanks in advance.
Julien Le DemWednesday, August 30th, 2023 at 12:23:18 PM GMT-04:00
I’ve seen people do this through the ingress controller in Kubernetes. Unfortunately I don’t have documentation besides k8s specific ones you would find for the ingress controller you’re using. You’d redirect any unauthenticated request to your identity provider
1
Michael RobinsonWednesday, August 30th, 2023 at 11:50:05 AM GMT-04:00
@channel
Friendly reminder: there’s a meetup tonight at Astronomer’s offices in SF!
1
Julien Le DemWednesday, August 30th, 2023 at 12:15:31 PM GMT-04:00
I’ll be there and looking forward to see @John Lukenoff ‘s presentation
Michael BarrientosWednesday, August 30th, 2023 at 9:38:31 PM GMT-04:00
Can anyone let 3 people stuck downstairs into the 7th floor?
👍1
Willy LulciucWednesday, August 30th, 2023 at 11:25:21 PM GMT-04:00
Sorry about that!
YunheThursday, August 31st, 2023 at 2:31:48 AM GMT-04:00
hello,everyone,i can run openLineage spark code in my notebook with python,but when use my idea to execute scala code like this:
import org.apache.spark.internal.Logging
import org.apache.spark.sql.SparkSession
import io.openlineage.client.OpenLineageClientUtils.loadOpenLineageYaml
import org.apache.spark.scheduler.{SparkListener, SparkListenerApplicationEnd, SparkListenerApplicationStart}
import sun.java2d.marlin.MarlinUtils.logInfo
object Test {
def main(args: Array[String]): Unit = {

val spark = SparkSession
.builder()
.master("local")
.appName("test")
.config("spark.jars.packages","io.openlineage:openlineage-spark:0.12.0")
.config("spark.extraListeners","io.openlineage.spark.agent.OpenLineageSparkListener")
.config("spark.openlineage.transport.type","console")
.getOrCreate()

spark.sparkContext.setLogLevel("INFO")

//spark.sparkContext.addSparkListener(new MySparkAppListener)
import spark.implicits._
val input = Seq((1, "zs", 2020), (2, "ls", 2023)).toDF("id", "name", "year")

input.select("id", "name").orderBy("id").show()

}

}

there is something wrong:
Exception in thread "spark-listener-group-shared" java.lang.NoSuchMethodError: io.openlineage.client.OpenLineageClientUtils.loadOpenLineageYaml(Ljava/io/InputStream;)Lio/openlineage/client/OpenLineageYaml;
at io.openlineage.spark.agent.ArgumentParser.extractOpenlineageConfFromSparkConf(ArgumentParser.java:114)
at io.openlineage.spark.agent.ArgumentParser.parse(ArgumentParser.java:78)
at io.openlineage.spark.agent.OpenLineageSparkListener.initializeContextFactoryIfNotInitialized(OpenLineageSparkListener.java:277)
at io.openlineage.spark.agent.OpenLineageSparkListener.onApplicationStart(OpenLineageSparkListener.java:267)
at org.apache.spark.scheduler.SparkListenerBus.doPostEvent(SparkListenerBus.scala:55)
at org.apache.spark.scheduler.SparkListenerBus.doPostEvent$(SparkListenerBus.scala:28)
at org.apache.spark.scheduler.AsyncEventQueue.doPostEvent(AsyncEventQueue.scala:37)
at org.apache.spark.scheduler.AsyncEventQueue.doPostEvent(AsyncEventQueue.scala:37)
at org.apache.spark.util.ListenerBus.postToAll(ListenerBus.scala:117)
at org.apache.spark.util.ListenerBus.postToAll$(ListenerBus.scala:101)
at org.apache.spark.scheduler.AsyncEventQueue.super$postToAll(AsyncEventQueue.scala:105)
at org.apache.spark.scheduler.AsyncEventQueue.$anonfun$dispatch$1(AsyncEventQueue.scala:105)
at scala.runtime.java8.JFunction0$mcJ$sp.apply(JFunction0$mcJ$sp.java:23)
at scala.util.DynamicVariable.withValue(DynamicVariable.scala:62)
at org.apache.spark.scheduler.AsyncEventQueue.org$apache$spark$scheduler$AsyncEventQueue$$dispatch(AsyncEventQueue.scala:100)
at org.apache.spark.scheduler.AsyncEventQueue$$anon$2.$anonfun$run$1(AsyncEventQueue.scala:96)
at org.apache.spark.util.Utils$.tryOrStopSparkContext(Utils.scala:1446)
at org.apache.spark.scheduler.AsyncEventQueue$$anon$2.run(AsyncEventQueue.scala:96)

i want to know how can i set idea scala environment correctly
Paweł LeszczyńskiThursday, August 31st, 2023 at 2:58:41 AM GMT-04:00
io.openlineage:openlineage-spark:0.12.0 -> could you repeat the steps with newer version?
YunheThursday, August 31st, 2023 at 3:51:52 AM GMT-04:00
ok,it`s my first use thie lineage tool. first,I added dependences in my pom.xml like this:
<dependency>
<groupId>io.openlineage</groupId>
<artifactId>openlineage-java</artifactId>
<version>0.12.0</version>
</dependency>
<dependency>
<groupId>org.apache.logging.log4j</groupId>
<artifactId>log4j-api</artifactId>
<version>2.7</version>
</dependency>
<dependency>
<groupId>org.apache.logging.log4j</groupId>
<artifactId>log4j-core</artifactId>
<version>2.7</version>
</dependency>
<dependency>
<groupId>org.apache.logging.log4j</groupId>
<artifactId>log4j-slf4j-impl</artifactId>
<version>2.7</version>
</dependency>
<dependency>
<groupId>io.openlineage</groupId>
<artifactId>openlineage-spark</artifactId>
<version>0.30.1</version>
</dependency>

my spark version is 3.3.1 and the version can not change

second, in file Openlineage/intergration/spark I enter command : docker-compose up and follow the steps in this doc:
https://openlineage.io/docs/integrations/spark/quickstart_local
there is no erro when i use notebook to execute pyspark for openlineage and I could get json message.
but after I enter "docker-compose up" ,I want to use my Idea tool to execute scala code like above,the erro happend like above. It seems that I does not configure the environment correctly. so how can i fix the problem .
Paweł LeszczyńskiFriday, September 1st, 2023 at 5:15:28 AM GMT-04:00
please use latest io.openlineage:openlineage-spark:1.1.0 instead. openlineage-java is already contained in the jar, no need to add it on your own.
Sheeri Cabral (Collibra)Thursday, August 31st, 2023 at 3:33:19 PM GMT-04:00
Will the August meeting be put up at https://wiki.lfaidata.foundation/display/OpenLineage/Monthly+TSC+meeting soon? (usually it’s up in a few days 🙂
Maciej ObuchowskiFriday, September 1st, 2023 at 6:00:53 AM GMT-04:00
@Michael Robinson
Michael RobinsonFriday, September 1st, 2023 at 5:13:32 PM GMT-04:00
The recording is on the youtube channel here. I’ll update the wiki ASAP
1
Julien Le DemThursday, August 31st, 2023 at 6:10:20 PM GMT-04:00
🎉13
🙌10
❤️5
Julien Le DemFriday, September 1st, 2023 at 11:09:55 PM GMT-04:00
Michael RobinsonFriday, September 1st, 2023 at 5:16:21 PM GMT-04:00
@channel
The latest issue of OpenLineage News is out now! Please subscribe to get it directly in your inbox each month.
🙌2
1
Anirudh ShrinivasonMonday, September 4th, 2023 at 3:38:28 AM GMT-04:00
Hi guys, I'd like to capture the spark.databricks.clusterUsageTags.clusterAllTags property from databricks. However, the value of this is a list of keys, and therefore cannot be supported by custom environment facet builder.
I was thinking that capturing this property might be useful for most databricks workloads, and whether it might make sense to auto-capture it along with other databricks variables, similar to how we capture mount points for the databricks jobs.
Does this sound okay? If so, then I can help to contribute this functionality
Maciej ObuchowskiMonday, September 4th, 2023 at 6:43:47 AM GMT-04:00
Sounds good to me
Anirudh ShrinivasonMonday, September 11th, 2023 at 5:15:03 AM GMT-04:00
Anirudh ShrinivasonMonday, September 4th, 2023 at 6:39:05 AM GMT-04:00
Also, another small clarification is that when using MergeIntoCommand, I'm receiving the lineage events on the backend, but I cannot seem to find any logging of the payload when I enable debug mode in openlineage. I remember there was a similar issue reported by another user in the past. May I check if it might be possible to help with this? It's making debugging quite hard for these cases. Thanks!
Maciej ObuchowskiMonday, September 4th, 2023 at 6:54:12 AM GMT-04:00
I think it only depends on log4j configuration
Maciej ObuchowskiMonday, September 4th, 2023 at 6:57:15 AM GMT-04:00
# Set everything to be logged to the console
 log4j.rootCategory=INFO, console
 log4j.appender.console=org.apache.log4j.ConsoleAppender
 log4j.appender.console.target=System.err
@@ -239,7 +193,7 @@
 &gt; Stopping at filesystem boundary (GIT_DISCOVERY_ACROSS_FILESYSTEM not set).
 &gt; fatal: not a git repository (or any parent up to mount point /home/airflow)
 &gt; Stopping at filesystem boundary (GIT_DISCOVERY_ACROSS_FILESYSTEM not set).

hm, this could be the actual bug?
Jakub DardzińskiWednesday, September 13th, 2023 at 10:06:51 AM GMT-04:00
that’s casual log in composer
Jakub DardzińskiWednesday, September 13th, 2023 at 10:12:16 AM GMT-04:00
extractor for &lt;class 'airflow.providers.google.cloud.operators.bigquery.BigQueryInsertJobOperator'&gt; is &lt;class 'big_query_insert_job_extractor.BigQueryInsertJobExtractor'

that’s actually class from your custom module, right?
Jakub DardzińskiWednesday, September 13th, 2023 at 10:14:03 AM GMT-04:00
I’ve done experiment, that’s how gcs looks like
Jakub DardzińskiWednesday, September 13th, 2023 at 10:14:09 AM GMT-04:00
and env vars
Jakub DardzińskiWednesday, September 13th, 2023 at 10:14:19 AM GMT-04:00
I have this extractor detected as expected
Jakub DardzińskiWednesday, September 13th, 2023 at 10:15:06 AM GMT-04:00
seens as &lt;class 'dependencies.bq.BigQueryInsertJobExtractor'&gt;
Jakub DardzińskiWednesday, September 13th, 2023 at 10:16:02 AM GMT-04:00
no __init__.py in base dags folder
Jakub DardzińskiWednesday, September 13th, 2023 at 10:17:02 AM GMT-04:00
I also checked that triggerer pod indeed has no gcsfuse set up, tbh no idea why, maybe some kind of optimization
the only effect is that when loading plugins in triggerer it throws some errors in logs, we don’t do anything at the moment there
Guntaka Jeevan PaulWednesday, September 13th, 2023 at 10:19:26 AM GMT-04:00
okk…got it @Jakub Dardziński…so the init at the top level of dags is as well not reqd, got it. Just one more doubt, there is a requirement where i want to change the operators property in the extractor inside the extract function, will that be taken into account and the operator’s execute be called with the property that i have populated in my extractor?
Guntaka Jeevan PaulWednesday, September 13th, 2023 at 10:21:28 AM GMT-04:00
for example i want to add a custom job_id to the BigQueryInsertJobOperator, so wheneerv someone uses the BigQueryInsertJobOperator operator i want to intercept that and add this job_id property to the operator…will that work?
Jakub DardzińskiWednesday, September 13th, 2023 at 10:24:46 AM GMT-04:00
I’m not sure if using OL for such thing is best choice. Wouldn’t it be better to subclass the operator?
Jakub DardzińskiWednesday, September 13th, 2023 at 10:25:37 AM GMT-04:00
but the answer is: it dependes on the airflow version, in 2.3+ I’m pretty sure the changed property stays in execute method
Guntaka Jeevan PaulWednesday, September 13th, 2023 at 10:27:49 AM GMT-04:00
yeah ideally that is how we should have done this but the problem is our client is having around 1000+ Dag’s in different google cloud projects, which are owned by multiple teams…so they are not willing to change anything in their dag. Thankfully they are using airflow 2.4.3
Maciej ObuchowskiWednesday, September 13th, 2023 at 10:31:15 AM GMT-04:00
1
Jakub DardzińskiWednesday, September 13th, 2023 at 10:35:30 AM GMT-04:00
btw I double-checked - execute method is in different process so this would not change task’s attribute there
Guntaka Jeevan PaulSaturday, September 16th, 2023 at 3:32:49 AM GMT-04:00
@Jakub Dardziński any idea how can we achieve this one. ---> https://openlineage.slack.com/archives/C01CK9T7HKR/p1694849427228709
Guntaka Jeevan PaulTuesday, September 12th, 2023 at 5:26:01 PM GMT-04:00
@here has anyone succeded in getting a custom extractor to work in GCP Cloud Composer or AWS MWAA, seems like there is no way
Mars LanTuesday, September 12th, 2023 at 5:34:29 PM GMT-04:00
Suraj GuptaWednesday, September 13th, 2023 at 1:44:27 AM GMT-04:00
I am exploring Spark - OpenLineage integration (using the latest PySpark and OL versions). I tested a simple pipeline which:
• Reads JSON data into PySpark DataFrame
• Apply data transformations
• Write transformed data to MySQL database
Observed that we receive 4 events (2 START and 2 COMPLETE) for the same job name. The events are almost identical with a small diff in the facets. All the events share the same runId, and we don't get any parentRunId.
Team, can you please confirm if this behaviour is expected? Seems to be different from the Airflow integration where we relate jobs to Parent Jobs.
Damien HawesWednesday, September 13th, 2023 at 2:54:37 AM GMT-04:00
The Spark integration requires that two parameters are passed to it, namely:

spark.openlineage.parentJobName
-spark.openlineage.parentRunId

You can find the list of parameters here:

https://github.com/OpenLineage/OpenLineage/blob/main/integration/spark/README.md
Suraj GuptaWednesday, September 13th, 2023 at 2:55:51 AM GMT-04:00
Thanks, will check this out
Damien HawesWednesday, September 13th, 2023 at 2:57:43 AM GMT-04:00
As for double accounting of events - that's a bit harder to diagnose.
Maciej ObuchowskiWednesday, September 13th, 2023 at 4:33:03 AM GMT-04:00
Can you share the the job and events?
Also @Paweł Leszczyński
Suraj GuptaWednesday, September 13th, 2023 at 6:03:49 AM GMT-04:00
Sure, sharing Job and events.
Suraj GuptaWednesday, September 13th, 2023 at 6:06:21 AM GMT-04:00
Paweł LeszczyńskiWednesday, September 13th, 2023 at 6:39:02 AM GMT-04:00
Hi @Suraj Gupta,

Thanks for providing such a detailed description of the problem.

It is not expected behaviour, it's an issue. The events correspond to the same logical plan which for some reason lead to sending two OL events. Is it reproducible aka. does it occur each time? If yes, we please feel free to raise an issue for that.

We have added in recent months several tests to verify amount of OL events being generated but we haven't tested it that way with JDBC. BTW. will the same happen if you write your data df_transformed to a file (like parquet file) ?
1
Suraj GuptaWednesday, September 13th, 2023 at 7:28:03 AM GMT-04:00
Thanks @Paweł Leszczyński, will confirm about writing to file and get back.
Suraj GuptaWednesday, September 13th, 2023 at 7:33:35 AM GMT-04:00
And yes, the issue is reproducible. Will raise an issue for this.
Paweł LeszczyńskiWednesday, September 13th, 2023 at 7:33:54 AM GMT-04:00
even if you write onto a file?
Suraj GuptaWednesday, September 13th, 2023 at 7:37:21 AM GMT-04:00
Yes, even when I write to a parquet file.
Paweł LeszczyńskiWednesday, September 13th, 2023 at 7:49:28 AM GMT-04:00
ok. i think i was able to reproduce it locally with https://github.com/OpenLineage/OpenLineage/pull/2103/files
Suraj GuptaWednesday, September 13th, 2023 at 7:56:11 AM GMT-04:00
Suraj GuptaMonday, September 25th, 2023 at 4:32:09 PM GMT-04:00
@Paweł Leszczyński I see that the PR is work in progress. Any rough estimate on when we can expect this fix to be released?
Paweł LeszczyńskiTuesday, September 26th, 2023 at 3:32:03 AM GMT-04:00
@Suraj Gupta put a comment within your issue. it's a bug we need to solve but I cannot bring any estimates today.
Suraj GuptaTuesday, September 26th, 2023 at 4:33:03 AM GMT-04:00
Thanks for update @Paweł Leszczyński, also please look into this comment. It might related and I'm not sure if expected behaviour.
Michael RobinsonWednesday, September 13th, 2023 at 2:20:32 PM GMT-04:00
@channel
This month’s TSC meeting, open to all, is tomorrow: https://openlineage.slack.com/archives/C01CK9T7HKR/p1694113940400549
1
Damien HawesThursday, September 14th, 2023 at 6:20:15 AM GMT-04:00
Context:

We use Spark with YARN, running on Hadoop 2.x (I can't remember the exact minor version) with Hive support.

Problem:

I'm noticed that CreateDataSourceAsSelectCommand objects are always transformed to an OutputDataset with a namespace value set to file - which is curious, because the inputs always have a (correct) namespace of hdfs://&lt;name-node&gt; - is this a known issue? A flaw with Apache Spark? A bug in the resolution logic?

For reference:

public class CreateDataSourceTableCommandVisitor
+spark.openlineage.parentRunId

You can find the list of parameters here:

https://github.com/OpenLineage/OpenLineage/blob/main/integration/spark/README.md
Suraj GuptaWednesday, September 13th, 2023 at 2:55:51 AM GMT-04:00
Thanks, will check this out
Damien HawesWednesday, September 13th, 2023 at 2:57:43 AM GMT-04:00
As for double accounting of events - that's a bit harder to diagnose.
Maciej ObuchowskiWednesday, September 13th, 2023 at 4:33:03 AM GMT-04:00
Can you share the the job and events?
Also @Paweł Leszczyński
Suraj GuptaWednesday, September 13th, 2023 at 6:03:49 AM GMT-04:00
Sure, sharing Job and events.
Suraj GuptaWednesday, September 13th, 2023 at 6:06:21 AM GMT-04:00
Paweł LeszczyńskiWednesday, September 13th, 2023 at 6:39:02 AM GMT-04:00
Hi @Suraj Gupta,

Thanks for providing such a detailed description of the problem.

It is not expected behaviour, it's an issue. The events correspond to the same logical plan which for some reason lead to sending two OL events. Is it reproducible aka. does it occur each time? If yes, we please feel free to raise an issue for that.

We have added in recent months several tests to verify amount of OL events being generated but we haven't tested it that way with JDBC. BTW. will the same happen if you write your data df_transformed to a file (like parquet file) ?
1
Suraj GuptaWednesday, September 13th, 2023 at 7:28:03 AM GMT-04:00
Thanks @Paweł Leszczyński, will confirm about writing to file and get back.
Suraj GuptaWednesday, September 13th, 2023 at 7:33:35 AM GMT-04:00
And yes, the issue is reproducible. Will raise an issue for this.
Paweł LeszczyńskiWednesday, September 13th, 2023 at 7:33:54 AM GMT-04:00
even if you write onto a file?
Suraj GuptaWednesday, September 13th, 2023 at 7:37:21 AM GMT-04:00
Yes, even when I write to a parquet file.
Paweł LeszczyńskiWednesday, September 13th, 2023 at 7:49:28 AM GMT-04:00
ok. i think i was able to reproduce it locally with https://github.com/OpenLineage/OpenLineage/pull/2103/files
Suraj GuptaWednesday, September 13th, 2023 at 7:56:11 AM GMT-04:00
Suraj GuptaMonday, September 25th, 2023 at 4:32:09 PM GMT-04:00
@Paweł Leszczyński I see that the PR is work in progress. Any rough estimate on when we can expect this fix to be released?
Paweł LeszczyńskiTuesday, September 26th, 2023 at 3:32:03 AM GMT-04:00
@Suraj Gupta put a comment within your issue. it's a bug we need to solve but I cannot bring any estimates today.
Suraj GuptaTuesday, September 26th, 2023 at 4:33:03 AM GMT-04:00
Thanks for update @Paweł Leszczyński, also please look into this comment. It might related and I'm not sure if expected behaviour.
Michael RobinsonWednesday, September 13th, 2023 at 2:20:32 PM GMT-04:00
@channel
This month’s TSC meeting, open to all, is tomorrow: https://openlineage.slack.com/archives/C01CK9T7HKR/p1694113940400549
1
Damien HawesThursday, September 14th, 2023 at 6:20:15 AM GMT-04:00
Context:

We use Spark with YARN, running on Hadoop 2.x (I can't remember the exact minor version) with Hive support.

Problem:

I'm noticed that CreateDataSourceAsSelectCommand objects are always transformed to an OutputDataset with a namespace value set to file - which is curious, because the inputs always have a (correct) namespace of hdfs://&lt;name-node&gt; - is this a known issue? A flaw with Apache Spark? A bug in the resolution logic?

For reference:

public class CreateDataSourceTableCommandVisitor
     extends QueryPlanVisitor&lt;CreateDataSourceTableCommand, OpenLineage.OutputDataset&gt; {
 
   public CreateDataSourceTableCommandVisitor(OpenLineageContext context) {
@@ -287,7 +241,7 @@
 Relation[id#3] orc
 
 == Physical Plan ==
-*(1) FileScan orc dhawes.ol_test_hadoop_src[id#3] Batched: true, Format: ORC, Location: InMemoryFileIndex[<hdfs://nn1/user/hive/warehouse/dhawes.db/ol_test_hadoop_src>], PartitionFilters: [], PushedFilters: [], ReadSchema: struct&lt;id:int&gt;

tatiThursday, September 14th, 2023 at 10:03:41 AM GMT-04:00
Hey everyone,
Any chance we could have a openlineage-integration-common 1.1.1 release with the following changes..?
https://github.com/OpenLineage/OpenLineage/pull/2106
https://github.com/OpenLineage/OpenLineage/pull/2108
6
tatiThursday, September 14th, 2023 at 10:05:19 AM GMT-04:00
Specially the first PR is affecting users of the astronomer-cosmos library: https://github.com/astronomer/astronomer-cosmos/issues/533
Michael RobinsonThursday, September 14th, 2023 at 10:05:24 AM GMT-04:00
Thanks @tati for requesting your first OpenLineage release! Three +1s from committers will authorize
1
Michael RobinsonThursday, September 14th, 2023 at 11:59:55 AM GMT-04:00
The release is authorized and will be initiated within two business days.
🎉1
tatiFriday, September 15th, 2023 at 4:40:12 AM GMT-04:00
Thanks a lot, @Michael Robinson!
Julien Le DemThursday, September 14th, 2023 at 8:23:01 PM GMT-04:00
Per discussion in the OpenLineage sync today here is a very early strawman proposal for an OpenLineage registry that producers and consumers could be registered in.
Feedback or alternate proposals welcome
https://docs.google.com/document/d/1zIxKST59q3I6ws896M4GkUn7IsueLw8ejct5E-TR0vY/edit
Once this is sufficiently fleshed out, I’ll create an actual proposal on github
👍1
Julien Le DemTuesday, October 3rd, 2023 at 8:33:35 PM GMT-04:00
I have cleaned up the registry proposal.
https://docs.google.com/document/d/1zIxKST59q3I6ws896M4GkUn7IsueLw8ejct5E-TR0vY/edit
In particular:
• I clarified that option 2 is preferred at this point.
• I moved discussion notes to the bottom. they will go away at some point
• Once it is stable, I’ll create a proposal with the preferred option.
• we need a good proposal for the core facets prefix. My suggestion is to move core facets to core in the registry. The drawback is prefix would be inconsistent.
Julien Le DemThursday, October 5th, 2023 at 5:34:12 PM GMT-04:00
I have created a ticket to make this easier to find. Once I get more feedback I’ll turn it into a md file in the repo: https://docs.google.com/document/d/1zIxKST59q3I6ws896M4GkUn7IsueLw8ejct5E-TR0vY/edit#heading=h.enpbmvu7n8gu
https://github.com/OpenLineage/OpenLineage/issues/2161
Michael RobinsonFriday, September 15th, 2023 at 12:03:27 PM GMT-04:00
@channel
Friendly reminder: the next OpenLineage meetup, our first in Toronto, is happening this coming Monday at 5 PM ET https://openlineage.slack.com/archives/C01CK9T7HKR/p1694441261486759
👍1
Guntaka Jeevan PaulSaturday, September 16th, 2023 at 3:30:27 AM GMT-04:00
@here we have dataproc operator getting called from a dag which submits a spark job, we wanted to maintain that continuity of parent job in the spark job so according to the documentation we can acheive that by using a macro called lineage_run_id that requires task and task_instance as the parameters. The problem we are facing is that our client’s have 1000's of dags, so asking them to change this everywhere it is used is not feasible, so we thought of using the task_policy feature in the airflow…but the problem is that task_policy gives you access to only the task/operator, but we don’t have the access to the task instance..that is required as a parameter to the lineage_run_id function. Can anyone kindly help us on how should we go about this one
t1 = DataProcPySparkOperator(
+*(1) FileScan orc dhawes.ol_test_hadoop_src[id#3] Batched: true, Format: ORC, Location: InMemoryFileIndex[<hdfs://nn1/user/hive/warehouse/dhawes.db/ol_test_hadoop_src>], PartitionFilters: [], PushedFilters: [], ReadSchema: struct&lt;id:int&gt;

tatiThursday, September 14th, 2023 at 10:03:41 AM GMT-04:00
Hey everyone,
Any chance we could have a openlineage-integration-common 1.1.1 release with the following changes..?
https://github.com/OpenLineage/OpenLineage/pull/2106
https://github.com/OpenLineage/OpenLineage/pull/2108
6
tatiThursday, September 14th, 2023 at 10:05:19 AM GMT-04:00
Specially the first PR is affecting users of the astronomer-cosmos library: https://github.com/astronomer/astronomer-cosmos/issues/533
Michael RobinsonThursday, September 14th, 2023 at 10:05:24 AM GMT-04:00
Thanks @tati for requesting your first OpenLineage release! Three +1s from committers will authorize
1
Michael RobinsonThursday, September 14th, 2023 at 11:59:55 AM GMT-04:00
The release is authorized and will be initiated within two business days.
🎉1
tatiFriday, September 15th, 2023 at 4:40:12 AM GMT-04:00
Thanks a lot, @Michael Robinson!
Julien Le DemThursday, September 14th, 2023 at 8:23:01 PM GMT-04:00
Per discussion in the OpenLineage sync today here is a very early strawman proposal for an OpenLineage registry that producers and consumers could be registered in.
Feedback or alternate proposals welcome
https://docs.google.com/document/d/1zIxKST59q3I6ws896M4GkUn7IsueLw8ejct5E-TR0vY/edit
Once this is sufficiently fleshed out, I’ll create an actual proposal on github
👍1
Julien Le DemTuesday, October 3rd, 2023 at 8:33:35 PM GMT-04:00
I have cleaned up the registry proposal.
https://docs.google.com/document/d/1zIxKST59q3I6ws896M4GkUn7IsueLw8ejct5E-TR0vY/edit
In particular:
• I clarified that option 2 is preferred at this point.
• I moved discussion notes to the bottom. they will go away at some point
• Once it is stable, I’ll create a proposal with the preferred option.
• we need a good proposal for the core facets prefix. My suggestion is to move core facets to core in the registry. The drawback is prefix would be inconsistent.
Julien Le DemThursday, October 5th, 2023 at 5:34:12 PM GMT-04:00
I have created a ticket to make this easier to find. Once I get more feedback I’ll turn it into a md file in the repo: https://docs.google.com/document/d/1zIxKST59q3I6ws896M4GkUn7IsueLw8ejct5E-TR0vY/edit#heading=h.enpbmvu7n8gu
https://github.com/OpenLineage/OpenLineage/issues/2161
Michael RobinsonFriday, September 15th, 2023 at 12:03:27 PM GMT-04:00
@channel
Friendly reminder: the next OpenLineage meetup, our first in Toronto, is happening this coming Monday at 5 PM ET https://openlineage.slack.com/archives/C01CK9T7HKR/p1694441261486759
👍1
Guntaka Jeevan PaulSaturday, September 16th, 2023 at 3:30:27 AM GMT-04:00
@here we have dataproc operator getting called from a dag which submits a spark job, we wanted to maintain that continuity of parent job in the spark job so according to the documentation we can acheive that by using a macro called lineage_run_id that requires task and task_instance as the parameters. The problem we are facing is that our client’s have 1000's of dags, so asking them to change this everywhere it is used is not feasible, so we thought of using the task_policy feature in the airflow…but the problem is that task_policy gives you access to only the task/operator, but we don’t have the access to the task instance..that is required as a parameter to the lineage_run_id function. Can anyone kindly help us on how should we go about this one
t1 = DataProcPySparkOperator(
     task_id=job_name,
     #required pyspark configuration,
     job_name=job_name,
@@ -302,7 +256,7 @@
     "rowCount": 1,
     "size": 0
    }
-}

I'm not sure what the size means here. Does this mean number of bytes inserted/updated?
Also, do we have any documentation for Spark specific Job and Run facets?
Paweł LeszczyńskiWednesday, September 27th, 2023 at 9:56:00 AM GMT-04:00
I am not sure it's stated in the doc. Here's the list of spark facets schemas: https://github.com/OpenLineage/OpenLineage/tree/main/integration/spark/shared/facets/spark/v1
Guntaka Jeevan PaulTuesday, September 26th, 2023 at 12:51:30 AM GMT-04:00
@here In Airflow Integration we send across a lineage Event for Dag start and complete, but that is not the case with spark integration…we don’t receive any event for the application start and complete in spark…is this expected behaviour or am i missing something?
1
Paweł LeszczyńskiWednesday, September 27th, 2023 at 9:47:39 AM GMT-04:00
For spark we do send start and complete for each spark action being run (single operation that causes spark processing being run). However, it is difficult for us to know if we're dealing with the last action within spark job or a spark script.
Paweł LeszczyńskiWednesday, September 27th, 2023 at 9:49:35 AM GMT-04:00
I think we need to look deeper into that as there is reoccuring need to capture such information
Paweł LeszczyńskiWednesday, September 27th, 2023 at 9:49:57 AM GMT-04:00
and spark listener event has methods like onApplicationStart and onApplicationEnd
Guntaka Jeevan PaulWednesday, September 27th, 2023 at 9:50:13 AM GMT-04:00
We are using the SparkListener, which has a function called OnApplicationStart which gets called whenever a spark application starts, so i was thinking why cant we send one at start and simlarly at end as well
Paweł LeszczyńskiWednesday, September 27th, 2023 at 9:50:33 AM GMT-04:00
additionally, we would like to have a concept of a parent run for a spark job which aggregates all actions run within a single spark job context
Guntaka Jeevan PaulWednesday, September 27th, 2023 at 9:51:11 AM GMT-04:00
yeah exactly. the way that it works with airflow integration
Paweł LeszczyńskiWednesday, September 27th, 2023 at 9:51:26 AM GMT-04:00
Paweł LeszczyńskiWednesday, September 27th, 2023 at 9:52:08 AM GMT-04:00
what you can is: come to our monthly Openlineage open meetings and raise that issue and convince the community about its importance
Guntaka Jeevan PaulWednesday, September 27th, 2023 at 9:53:32 AM GMT-04:00
yeah sure would love to do that…how can i join them, will that be posted here in this slack channel?
Michael RobinsonWednesday, September 27th, 2023 at 9:54:08 AM GMT-04:00
Hi, you can see the schedule and RSVP here: https://openlineage.io/community
🙌1
1
Michael RobinsonWednesday, September 27th, 2023 at 11:19:16 AM GMT-04:00
Meetup recap: Toronto Meetup @ Airflow Summit, September 18, 2023
It was great to see so many members of our community at this event! I counted 32 total attendees, with all but a handful being first-timers.
Topics included:
• Presentation on the history, architecture and roadmap of the project by @Julien Le Dem and @Harel Shein
• Discussion of OpenLineage support in Marquez by @Willy Lulciuc
• Presentation by Ye Liu and Ivan Perepelitca from Metaphor, the social platform for data, about their integration
• Presentation by @Paweł Leszczyński about the Spark integration
• Presentation by @Maciej Obuchowski about the Apache Airflow Provider
Thanks to all the presenters and attendees with a shout out to @Harel Shein for the help with organizing and day-of logistics, @Jakub Dardziński for the help with set up/clean up, and @Sheeri Cabral (Collibra) for the crucial assist with the signup sheet.
This was our first meetup in Toronto, and we learned some valuable lessons about planning events in new cities — the first and foremost being to ask for a pic of the building! 🙂 But it seemed like folks were undeterred, and the space itself lived up to expectations.
For a recording and clips from the meetup, head over to our YouTube channel.
Upcoming events:
• October 5th in San Francisco: Marquez Meetup @ Astronomer (sign up here)
• November: Warsaw meetup (details, date TBA)
• January: London meetup (details, date TBA)
Are you interested in hosting or co-hosting an OpenLineage or Marquez meetup? DM me!
🙌3
❤️6
🚀2
😅1
1
Michael RobinsonWednesday, September 27th, 2023 at 11:55:47 AM GMT-04:00
A few more pics:
Damien HawesWednesday, September 27th, 2023 at 12:23:05 PM GMT-04:00
Hi folks, am I correct in my observations that the Spark integration does not generate inputs and outputs for Kafka-to-Kafka pipelines?

EDIT: Removed the crazy wall of text. Relevant GitHub issue is here.
👀1
Paweł LeszczyńskiThursday, September 28th, 2023 at 2:42:18 AM GMT-04:00
responded within the issue
Erik AlfthanThursday, September 28th, 2023 at 2:40:40 AM GMT-04:00
Hello community
First time poster - bear with me :)

I am looking to make a minor PR on the airflow integration (fixing github #2130), and the code change is easy enough, but I fail to install the python environment. I have tried the simple ones
OpenLineage/integration/airflow &gt; pip install -e .
or
OpenLineage/integration/airflow &gt; pip install -r dev-requirements.txt
but they both fail on
ERROR: No matching distribution found for openlineage-sql==1.3.0

(which I think is an unreleased version in the git project)

How would I go about to install the requirements?

//Erik

PS. Sorry for posting this in general if there is a specific integration or contribution channel - I didnt find a better channel
Paweł LeszczyńskiThursday, September 28th, 2023 at 3:04:48 AM GMT-04:00
Hi @Erik Alfthan, the channel is totally OK. I am not airflow integration expert, but what it looks to me, you're missing openlineage-sql library, which is a rust library used to extract lineage from sql queries. This is how we do that in circle ci:
https://app.circleci.com/pipelines/github/OpenLineage/OpenLineage/8080/workflows/aba53369-836c-48f5-a2dd-51bc0740a31c/jobs/140113

and subproject page with build instructions: https://github.com/OpenLineage/OpenLineage/tree/main/integration/sql
Erik AlfthanThursday, September 28th, 2023 at 3:07:23 AM GMT-04:00
Ok, so I go and "manually" build the internal dependency so that it becomes available in the pip cache?

I was hoping for something more automagical, but that should work
Paweł LeszczyńskiThursday, September 28th, 2023 at 3:08:06 AM GMT-04:00
I think so. @Jakub Dardziński am I right?
Jakub DardzińskiThursday, September 28th, 2023 at 3:18:27 AM GMT-04:00
https://openlineage.io/docs/development/developing/python/setup
there’s a guide how to setup the dev environment

> Typically, you first need to build openlineage-sql locally (see README). After each release you have to repeat this step in order to bump local version of the package.
This might be somewhat exposed more in GitHub repository README as well
Erik AlfthanThursday, September 28th, 2023 at 3:27:20 AM GMT-04:00
It didnt find the wheel in the cache, but if I used the line in the sql/README.md
pip install openlineage-sql --no-index --find-links ../target/wheels --force-reinstall
It is installed and thus skipped/passed when pip later checks if it needs to be installed.

Now I have a second issue because it is expecting me to have mysqlclient-2.2.0 which seems to need a binary
Command 'pkg-config --exists mysqlclient' returned non-zero exit status 127
and
Command 'pkg-config --exists mariadb' returned non-zero exit status 127
I am on Ubuntu 22.04 in WSL2. Should I go to apt and grab me a mysql client?
Jakub DardzińskiThursday, September 28th, 2023 at 3:31:52 AM GMT-04:00
> It didnt find the wheel in the cache, but if I used the line in the sql/README.md
> pip install openlineage-sql --no-index --find-links ../target/wheels --force-reinstall
> It is installed and thus skipped/passed when pip later checks if it needs to be installed.
That’s actually expected. You should build new wheel locally and then install it.

> Now I have a second issue because it is expecting me to have mysqlclient-2.2.0 which seems to need a binary
> Command 'pkg-config --exists mysqlclient' returned non-zero exit status 127
> and
> Command 'pkg-config --exists mariadb' returned non-zero exit status 127
> I am on Ubuntu 22.04 in WSL2. Should I go to apt and grab me a mysql client?
We’ve left some system specific configuration, e.g. mysqlclient, to users as it’s a bit aside from OpenLineage and more of general development task.

probably
sudo apt-get install python3-dev default-libmysqlclient-dev build-essential 

should work
Erik AlfthanThursday, September 28th, 2023 at 3:32:04 AM GMT-04:00
I just realized that I should probably skip setting up my wsl and just run the tests in the docker setup you prepared
Jakub DardzińskiThursday, September 28th, 2023 at 3:35:46 AM GMT-04:00
You could do that as well but if you want to test your changes vs many Airflow versions that wouldn’t be possible I think (run them with tox btw)
Erik AlfthanThursday, September 28th, 2023 at 4:54:39 AM GMT-04:00
This is starting to feel like a rabbit hole 😞

When I run tox, I get a lot of build errors
• client needs to be built
• sql needs to be built to a different target than its readme says
• a lot of builds fail on cython_sources
Jakub DardzińskiThursday, September 28th, 2023 at 5:19:34 AM GMT-04:00
would you like to share some exact log lines? I’ve never seen such errors, they probably are system specific
Erik AlfthanThursday, September 28th, 2023 at 6:45:48 AM GMT-04:00
Getting requirements to build wheel did not run successfully.
│ exit code: 1
╰─&gt; [62 lines of output]
/tmp/pip-build-env-q1pay0xo/overlay/lib/python3.10/site-packages/setuptools/config/setupcfg.py:293: _DeprecatedConfig: Deprecated config in setup.cfg` +}
I'm not sure what the size means here. Does this mean number of bytes inserted/updated?
Also, do we have any documentation for Spark specific Job and Run facets?
Paweł LeszczyńskiWednesday, September 27th, 2023 at 9:56:00 AM GMT-04:00
I am not sure it's stated in the doc. Here's the list of spark facets schemas: https://github.com/OpenLineage/OpenLineage/tree/main/integration/spark/shared/facets/spark/v1
Guntaka Jeevan PaulTuesday, September 26th, 2023 at 12:51:30 AM GMT-04:00
@here In Airflow Integration we send across a lineage Event for Dag start and complete, but that is not the case with spark integration…we don’t receive any event for the application start and complete in spark…is this expected behaviour or am i missing something?
1
Paweł LeszczyńskiWednesday, September 27th, 2023 at 9:47:39 AM GMT-04:00
For spark we do send start and complete for each spark action being run (single operation that causes spark processing being run). However, it is difficult for us to know if we're dealing with the last action within spark job or a spark script.
Paweł LeszczyńskiWednesday, September 27th, 2023 at 9:49:35 AM GMT-04:00
I think we need to look deeper into that as there is reoccuring need to capture such information
Paweł LeszczyńskiWednesday, September 27th, 2023 at 9:49:57 AM GMT-04:00
and spark listener event has methods like onApplicationStart and onApplicationEnd
Guntaka Jeevan PaulWednesday, September 27th, 2023 at 9:50:13 AM GMT-04:00
We are using the SparkListener, which has a function called OnApplicationStart which gets called whenever a spark application starts, so i was thinking why cant we send one at start and simlarly at end as well
Paweł LeszczyńskiWednesday, September 27th, 2023 at 9:50:33 AM GMT-04:00
additionally, we would like to have a concept of a parent run for a spark job which aggregates all actions run within a single spark job context
Guntaka Jeevan PaulWednesday, September 27th, 2023 at 9:51:11 AM GMT-04:00
yeah exactly. the way that it works with airflow integration
Paweł LeszczyńskiWednesday, September 27th, 2023 at 9:51:26 AM GMT-04:00
Paweł LeszczyńskiWednesday, September 27th, 2023 at 9:52:08 AM GMT-04:00
what you can is: come to our monthly Openlineage open meetings and raise that issue and convince the community about its importance
Guntaka Jeevan PaulWednesday, September 27th, 2023 at 9:53:32 AM GMT-04:00
yeah sure would love to do that…how can i join them, will that be posted here in this slack channel?
Michael RobinsonWednesday, September 27th, 2023 at 9:54:08 AM GMT-04:00
Hi, you can see the schedule and RSVP here: https://openlineage.io/community
🙌1
1
Michael RobinsonWednesday, September 27th, 2023 at 11:19:16 AM GMT-04:00
Meetup recap: Toronto Meetup @ Airflow Summit, September 18, 2023
It was great to see so many members of our community at this event! I counted 32 total attendees, with all but a handful being first-timers.
Topics included:
• Presentation on the history, architecture and roadmap of the project by @Julien Le Dem and @Harel Shein
• Discussion of OpenLineage support in Marquez by @Willy Lulciuc
• Presentation by Ye Liu and Ivan Perepelitca from Metaphor, the social platform for data, about their integration
• Presentation by @Paweł Leszczyński about the Spark integration
• Presentation by @Maciej Obuchowski about the Apache Airflow Provider
Thanks to all the presenters and attendees with a shout out to @Harel Shein for the help with organizing and day-of logistics, @Jakub Dardziński for the help with set up/clean up, and @Sheeri Cabral (Collibra) for the crucial assist with the signup sheet.
This was our first meetup in Toronto, and we learned some valuable lessons about planning events in new cities — the first and foremost being to ask for a pic of the building! 🙂 But it seemed like folks were undeterred, and the space itself lived up to expectations.
For a recording and clips from the meetup, head over to our YouTube channel.
Upcoming events:
• October 5th in San Francisco: Marquez Meetup @ Astronomer (sign up here)
• November: Warsaw meetup (details, date TBA)
• January: London meetup (details, date TBA)
Are you interested in hosting or co-hosting an OpenLineage or Marquez meetup? DM me!
🙌3
❤️6
🚀2
😅1
1
Michael RobinsonWednesday, September 27th, 2023 at 11:55:47 AM GMT-04:00
A few more pics:
Damien HawesWednesday, September 27th, 2023 at 12:23:05 PM GMT-04:00
Hi folks, am I correct in my observations that the Spark integration does not generate inputs and outputs for Kafka-to-Kafka pipelines?

EDIT: Removed the crazy wall of text. Relevant GitHub issue is here.
👀1
Paweł LeszczyńskiThursday, September 28th, 2023 at 2:42:18 AM GMT-04:00
responded within the issue
Erik AlfthanThursday, September 28th, 2023 at 2:40:40 AM GMT-04:00
Hello community
First time poster - bear with me :)

I am looking to make a minor PR on the airflow integration (fixing github #2130), and the code change is easy enough, but I fail to install the python environment. I have tried the simple ones
OpenLineage/integration/airflow &gt; pip install -e .
or
OpenLineage/integration/airflow &gt; pip install -r dev-requirements.txt
but they both fail on
ERROR: No matching distribution found for openlineage-sql==1.3.0

(which I think is an unreleased version in the git project)

How would I go about to install the requirements?

//Erik

PS. Sorry for posting this in general if there is a specific integration or contribution channel - I didnt find a better channel
Paweł LeszczyńskiThursday, September 28th, 2023 at 3:04:48 AM GMT-04:00
Hi @Erik Alfthan, the channel is totally OK. I am not airflow integration expert, but what it looks to me, you're missing openlineage-sql library, which is a rust library used to extract lineage from sql queries. This is how we do that in circle ci:
https://app.circleci.com/pipelines/github/OpenLineage/OpenLineage/8080/workflows/aba53369-836c-48f5-a2dd-51bc0740a31c/jobs/140113

and subproject page with build instructions: https://github.com/OpenLineage/OpenLineage/tree/main/integration/sql
Erik AlfthanThursday, September 28th, 2023 at 3:07:23 AM GMT-04:00
Ok, so I go and "manually" build the internal dependency so that it becomes available in the pip cache?

I was hoping for something more automagical, but that should work
Paweł LeszczyńskiThursday, September 28th, 2023 at 3:08:06 AM GMT-04:00
I think so. @Jakub Dardziński am I right?
Jakub DardzińskiThursday, September 28th, 2023 at 3:18:27 AM GMT-04:00
https://openlineage.io/docs/development/developing/python/setup
there’s a guide how to setup the dev environment

> Typically, you first need to build openlineage-sql locally (see README). After each release you have to repeat this step in order to bump local version of the package.
This might be somewhat exposed more in GitHub repository README as well
Erik AlfthanThursday, September 28th, 2023 at 3:27:20 AM GMT-04:00
It didnt find the wheel in the cache, but if I used the line in the sql/README.md
pip install openlineage-sql --no-index --find-links ../target/wheels --force-reinstall
It is installed and thus skipped/passed when pip later checks if it needs to be installed.

Now I have a second issue because it is expecting me to have mysqlclient-2.2.0 which seems to need a binary
Command 'pkg-config --exists mysqlclient' returned non-zero exit status 127
and
Command 'pkg-config --exists mariadb' returned non-zero exit status 127
I am on Ubuntu 22.04 in WSL2. Should I go to apt and grab me a mysql client?
Jakub DardzińskiThursday, September 28th, 2023 at 3:31:52 AM GMT-04:00
> It didnt find the wheel in the cache, but if I used the line in the sql/README.md
> pip install openlineage-sql --no-index --find-links ../target/wheels --force-reinstall
> It is installed and thus skipped/passed when pip later checks if it needs to be installed.
That’s actually expected. You should build new wheel locally and then install it.

> Now I have a second issue because it is expecting me to have mysqlclient-2.2.0 which seems to need a binary
> Command 'pkg-config --exists mysqlclient' returned non-zero exit status 127
> and
> Command 'pkg-config --exists mariadb' returned non-zero exit status 127
> I am on Ubuntu 22.04 in WSL2. Should I go to apt and grab me a mysql client?
We’ve left some system specific configuration, e.g. mysqlclient, to users as it’s a bit aside from OpenLineage and more of general development task.

probably
sudo apt-get install python3-dev default-libmysqlclient-dev build-essential 

should work
Erik AlfthanThursday, September 28th, 2023 at 3:32:04 AM GMT-04:00
I just realized that I should probably skip setting up my wsl and just run the tests in the docker setup you prepared
Jakub DardzińskiThursday, September 28th, 2023 at 3:35:46 AM GMT-04:00
You could do that as well but if you want to test your changes vs many Airflow versions that wouldn’t be possible I think (run them with tox btw)
Erik AlfthanThursday, September 28th, 2023 at 4:54:39 AM GMT-04:00
This is starting to feel like a rabbit hole 😞

When I run tox, I get a lot of build errors
• client needs to be built
• sql needs to be built to a different target than its readme says
• a lot of builds fail on cython_sources
Jakub DardzińskiThursday, September 28th, 2023 at 5:19:34 AM GMT-04:00
would you like to share some exact log lines? I’ve never seen such errors, they probably are system specific
Erik AlfthanThursday, September 28th, 2023 at 6:45:48 AM GMT-04:00
Getting requirements to build wheel did not run successfully.
│ exit code: 1
╰─&gt; [62 lines of output]
/tmp/pip-build-env-q1pay0xo/overlay/lib/python3.10/site-packages/setuptools/config/setupcfg.py:293: _DeprecatedConfig: Deprecated config in setup.cfg` !! ** @@ -671,7 +625,68 @@ )
Mike FangFriday, November 3rd, 2023 at 12:52:55 PM GMT-04:00
ok great thanks for following up on this, super helpful
Michael RobinsonThursday, November 2nd, 2023 at 12:00:00 PM GMT-04:00
👍5
🚀1
Jason YipThursday, November 2nd, 2023 at 2:49:18 PM GMT-04:00
@Paweł Leszczyński I tested 1.5.0, it works great now, but the environment facets is gone in START... which I very much want it.. any thoughts?
Jason YipFriday, November 3rd, 2023 at 4:18:11 AM GMT-04:00
actually, it shows up in one of the RUNNING now... behavior is consistent between 11.3 and 13.3, thanks for fixing this issue
👍1
Jason YipSaturday, November 4th, 2023 at 3:44:22 PM GMT-04:00
@Paweł Leszczyński looks like I need to bring bad news.. 13.3 is fixed for specific scenarios, but 11.3 is still reading output as dbfs.. there are scenarios that it's not producing input and output like:

create table table using delta as
location 'abfss://....'
Select * from parquet.`abfss://....'
Jason YipSaturday, November 4th, 2023 at 3:44:31 PM GMT-04:00
Will test more and ope issues
Rodrigo MaiaMonday, November 6th, 2023 at 5:34:33 AM GMT-05:00
@Jason Yiphow did you manage the get the environment attribute. it's not showing up to me at all. I've tried databricks abut also tried a local instance of spark.
Jason YipTuesday, November 7th, 2023 at 6:32:02 PM GMT-05:00
@Rodrigo Maia its showing up in one of the RUNNING events, not in the START event anymore
Rodrigo MaiaWednesday, November 8th, 2023 at 3:04:32 AM GMT-05:00
I never had a running event :melting_face: Am I filtering something?
Jason YipWednesday, November 8th, 2023 at 1:03:26 PM GMT-05:00
Umm.. ok show me your code, will try on my end
Jason YipWednesday, November 8th, 2023 at 2:26:06 PM GMT-05:00
@Paweł Leszczyński @Rodrigo Maia actually if you are using UC-enabled cluster, you won't get any RUNNING events
Michael RobinsonFriday, November 3rd, 2023 at 12:00:07 PM GMT-04:00
@channel
This month’s TSC meeting (open to all) is next Thursday the 9th at 10am PT. On the agenda:
• announcements
• recent releases
• recent additions to the Flink integration by @U05QA2D1XNV
• recent additions to the Spark integration by @Paweł Leszczyński
• updates on proposals by @Julien Le Dem
• discussion topics
• open discussion
More info and the meeting link can be found on the website. All are welcome! Do you have a discussion topic, use case or integration you’d like to demo? DM me to be added to the agenda.
👍1
priya narayanaSaturday, November 4th, 2023 at 7:08:10 AM GMT-04:00
Hi Team , we are trying to customize the events by writing custom lineage listener extending OpenLineageSparkListener, but would need some direction how to capture the events
Jakub DardzińskiSaturday, November 4th, 2023 at 7:11:46 AM GMT-04:00
priya narayanaSaturday, November 4th, 2023 at 7:13:47 AM GMT-04:00
yes
Jakub DardzińskiSaturday, November 4th, 2023 at 7:15:21 AM GMT-04:00
It seems pretty extensively described, what kind of help do you need?
priya narayanaSaturday, November 4th, 2023 at 7:16:13 AM GMT-04:00
io.openlineage.spark.api.OpenLineageEventHandlerFactory if i use this how will i pass custom listener to my spark submit
priya narayanaSaturday, November 4th, 2023 at 7:17:25 AM GMT-04:00
I would like to know how will i customize my events using this . For example: - In "input" Facet i want only symlinks name i am not intereseted in anything else
priya narayanaSaturday, November 4th, 2023 at 7:17:32 AM GMT-04:00
can you please provide some guidance
priya narayanaSaturday, November 4th, 2023 at 7:18:36 AM GMT-04:00
@Jakub Dardziński this is the doubt i have
priya narayanaSaturday, November 4th, 2023 at 8:17:25 AM GMT-04:00
Some one who did spark integration throw some light
Jakub DardzińskiSaturday, November 4th, 2023 at 8:21:22 AM GMT-04:00
it's weekend for most of us so you probably need to wait until Monday for precise answers
David GossMonday, November 6th, 2023 at 4:03:42 AM GMT-05:00
👋 I raised a PR https://github.com/OpenLineage/OpenLineage/pull/2223 off the back of some Marquez conversations a while back to try and clarify how names of Snowflake objects should be expressed in OL events. I used Snowflake’s OL view as a guide, but also I appreciate there are other OL producers that involve Snowflake too (Airflow? dbt?). Any feedback on this would be appreciated!
David GossWednesday, November 8th, 2023 at 10:42:35 AM GMT-05:00
Thanks for merging this @Maciej Obuchowski!
👍1
Athitya KumarMonday, November 6th, 2023 at 5:22:03 AM GMT-05:00
Hey team! 👋

We're trying to use openlineage-flink, and would like provide the openlineage.transport.type=http and configure other transport configs, but we're not able to find sufficient docs (tried this doc) on where/how these configs can be provided.

For example, in spark, the changes mostly were delegated to the spark-submit command like
spark-submit --conf "spark.extraListeners=io.openlineage.spark.agent.OpenLineageSparkListener" \
     --packages "io.openlineage:openlineage-spark:&lt;spark-openlineage-version&gt;" \
     --conf "spark.openlineage.transport.url=http://{openlineage.client.host}/api/v1/namespaces/spark_integration/" \
-    --class com.mycompany.MySparkApp my_application.jar

And the OpenLineageSparkListener has a method to retrieve the provided spark confs as an object in the ArgumentParser. Similarly, looking for some pointers on how the openlineage.transport configs can be provided to OpenLineageFlinkJobListener & how the flink listener parses/uses these configs

TIA! 😄
Maciej ObuchowskiTuesday, November 7th, 2023 at 5:56:09 AM GMT-05:00
similarly to spark config, you can use flink config
Athitya KumarTuesday, November 7th, 2023 at 10:36:53 PM GMT-05:00
@Maciej Obuchowski - Got it. Our use-case is that we're trying to build a wrapper on top of openlineage-flink for productionising for our flink jobs.

We're trying to have a wrapper class that extends OpenLineageFlinkJobListener class, and overwrites the HTTP transport endpoint/url to a constant value (say, example.com and /api/v1/flink). But we see that the OpenLineageFlinkJobListener constructor is defined as a private constructor - just wanted to check with the team whether it was just a default scope, or intended to be private. If it was just a default scope, can we contribute a PR to make it public, to make it friendly for teams trying to adopt & extend openlineage?

And also, we wanted to understand better on where we're reading the HTTP transport endpoint/url configs in OpenLineageFlinkJobListener and what'd be the best place to override it to the constant endpoint/url for our use-case
Maciej ObuchowskiWednesday, November 8th, 2023 at 5:55:43 AM GMT-05:00
We parse flink conf to get that information: https://github.com/OpenLineage/OpenLineage/blob/26494b596e9669d2ada164066a73c44e04[…]ink/src/main/java/io/openlineage/flink/client/EventEmitter.java

> But we see that the OpenLineageFlinkJobListener constructor is defined as a private constructor - just wanted to check with the team whether it was just a default scope, or intended to be private.
The way to construct is is a public builder in the same class

I think easier way than wrapper class would be use existing flink configuration, or to set up OPENLINEAGE_URL env variable, or have openlineage.yml config file - not sure why this is the way you've chosen?
Athitya KumarThursday, November 9th, 2023 at 12:41:02 PM GMT-05:00
I think easier way than wrapper class would be use existing flink configuration, or to set up OPENLINEAGE_URL env variable, or have openlineage.yml config file - not sure why this is the way you've chosen?
@Maciej Obuchowski - The reasoning behind going with a wrapper class is that we can abstract out the nitty-gritty like how/where we're publishing openlineage events etc - especially for companies that have a lot of teams that may be adopting openlineage.

For example, if we wanna move away from http transport to kafka transport - we'd be changing only this wrapper class and ask folks to update their wrapper class dependency version. If we went without the wrapper class, then the exact config changes would need to be synced and done by many different teams, who may not have enough context.

Similarly, if we wanna enable some other default best-practise configs, or inject any company-specific configs etc, the wrapper would be useful in abstracting out the details and be the 1 place that handles all openlineage related integrations for any future changes.

That's why we wanna extend openlineage's listener class & leverage most of the OSS code as-is; and at the same time, have the ability to extend & inject customisations. I think that's where some things like having getters for the class object attributes, or having public constructors would be really helpful 😄
Maciej ObuchowskiThursday, November 9th, 2023 at 1:03:56 PM GMT-05:00
@Athitya Kumar that makes sense. Feel free to provide PR adding getters and stuff.
🎉1
Yannick LibertTuesday, November 7th, 2023 at 6:03:49 AM GMT-05:00
Hi all, we (I work with @U05VDHJJ9T7 and @Abdallah) have a quick question regarding the spark integration:
if a spark app contains several jobs, they will be named "my_spark_app_name.job1" and "my_spark_app_name.job2"
eg:
spark_job.collect_limit
spark_job.map_partitions_parallel_collection

If I understood correctly, the spark integration maps one Spark job to a single OpenLineage Job, and the application itself should be assigned a Run id at startup and each job that executes will report the application's Run id as its parent job run (taken from: https://openlineage.io/docs/integrations/spark/).

In our case, the app Run Id is never created, and the jobs runs don't contain any parent facets. We tested it with a recent integration version in 1.4.1 and also an older one (0.26.0).
Did we miss something in the OL spark integration config?
Paweł LeszczyńskiTuesday, November 7th, 2023 at 6:07:51 AM GMT-05:00
hey, a name of the output dataset should be put at the end of the job name. This was introduced to help with jobs that call multiple spark actions
Yannick LibertTuesday, November 7th, 2023 at 7:05:52 AM GMT-05:00
Hi Paweł,
Thanks for your answer, yes indeed with the newer version of OL, we automatically have the name of the output dataset at the end of the job name, but no App run id, nor any parent run facet.
Paweł LeszczyńskiTuesday, November 7th, 2023 at 8:16:44 AM GMT-05:00
yes, you're right. I mean you can set in config spark.openlineage.parentJobName which will be shared through whole app run, but this needs to be set manually
Yannick LibertTuesday, November 7th, 2023 at 8:36:58 AM GMT-05:00
I see, thanks a lot for your reply we'll try that
ldaceyTuesday, November 7th, 2023 at 10:49:25 AM GMT-05:00
if I have a dataset on adls gen2 which synapse connects to as an external delta table, is that the use case of a symlink dataset? the delta table is connected to by PBI and by Synapse, but the underlying data is exactly the same
Maciej ObuchowskiWednesday, November 8th, 2023 at 10:49:04 AM GMT-05:00
Sounds like it, yes - if the logical dataset names are different but physical one is the same
Rodrigo MaiaWednesday, November 8th, 2023 at 12:38:52 PM GMT-05:00
Has anyone here tried OpenLineage with Spark on Amazon EMR?
Jason YipWednesday, November 8th, 2023 at 1:01:16 PM GMT-05:00
No but it should work the same I tried on AWS and Google Colab and Azure
👍1
U053LCT71BQThursday, November 9th, 2023 at 3:10:54 AM GMT-05:00
Yes. @Abdallah could provide some details if needed.
👍1
🔥1
Michael RobinsonWednesday, November 8th, 2023 at 12:44:54 PM GMT-05:00
@channel
Friendly reminder: this month’s TSC meeting, open to all, is tomorrow at 10 am PT: https://openlineage.slack.com/archives/C01CK9T7HKR/p1699027207361229
👍1
Jason YipFriday, November 10th, 2023 at 3:25:45 PM GMT-05:00
@Paweł Leszczyński regarding to https://github.com/OpenLineage/OpenLineage/issues/2124, OL is parsing out the table location in Hive metastore, it is the location of the table in the catalog and not the physical location of the data. It is both right and wrong because it is a table, just it is an external table.

https://docs.databricks.com/en/sql/language-manual/sql-ref-external-tables.html
Jason YipFriday, November 10th, 2023 at 3:32:28 PM GMT-05:00
Jason YipSaturday, November 11th, 2023 at 3:29:33 AM GMT-05:00
@Paweł Leszczyński this is why if create a table with adls location it won't show input and output:

https://github.com/OpenLineage/OpenLineage/blob/main/integration/spark/spark35/src[…]k35/agent/lifecycle/plan/CreateReplaceOutputDatasetBuilder.java

Because the catalog object is not there.
Jason YipSaturday, November 11th, 2023 at 3:30:44 AM GMT-05:00
Databricks needs to be re-written in a way that supports Databricks it seems like
Jason YipMonday, November 13th, 2023 at 3:00:42 AM GMT-05:00
@Paweł Leszczyński I went back to 1.4.1, output does show adls location. But environment facet is gone in 1.4.1. It shows up in 1.5.0 but namespace is back to dbfs....
Jason YipMonday, November 13th, 2023 at 3:18:37 AM GMT-05:00
@Paweł Leszczyński I diff CreateReplaceDatasetBuilder.java and CreateReplaceOutputDatasetBuilder.java and they are the same except for the class name, so I am not sure what is causing the change. I also realize you don't have a test case for ADLS
Paweł LeszczyńskiMonday, November 13th, 2023 at 4:52:07 AM GMT-05:00
Thanks @Jason Yip for your engagement in finding the cause and solution to this issue.

Among the technical problems, another problem here is that our databricks integration tests are run on AWS and the issue you describe occurs in Azure. I would consider this a primary issue as it is difficult for me to verify the behaviour you describe and fix it with a failing integration test at the start.

Are you able to reproduce the issue on AWS Databricks environment so that we could include it in our integration tests and make sure the behvaiour will not change later on in future?
Jason YipMonday, November 13th, 2023 at 6:06:44 PM GMT-05:00
I didn't know Azure and AWS Databricks are different. Let me try it on AWS as well
Naresh reddyWednesday, November 15th, 2023 at 7:17:24 AM GMT-05:00
Hi
Can anyone point me to the deck on how Airflow can be integrated using Openlineage?
Naresh reddyWednesday, November 15th, 2023 at 7:27:55 AM GMT-05:00
thank you @Maciej Obuchowski
Naresh reddyWednesday, November 15th, 2023 at 11:09:24 AM GMT-05:00
Can anyone tell me why OL is better than other competitors if you can provide an analysis that would be great
Harel SheinThursday, November 16th, 2023 at 11:46:16 AM GMT-05:00
Hey @Naresh reddy can you help me understand what you mean by competitors?
OL is a specification that can be used to solve various problems, so if you have a clear problem statement, maybe I can help with pros/cons for that problem
Naresh reddyWednesday, November 15th, 2023 at 11:10:58 AM GMT-05:00
what are the pros and cons of OL. we often talk about positives to market it but what are the pain points using OL,how it's addressing user issues?
Michael RobinsonThursday, November 16th, 2023 at 1:38:42 PM GMT-05:00
Hi @Naresh reddy, thanks for your question. We’ve heard that OpenLineage is attractive because of its desirable integrations, including a best-in-class Spark integration, its extensibility, the fact that it’s not destructive, and the fact that it’s open source. I’m not aware of pain points per se, but there are certainly features and integrations that we wish we could focus on but can’t at the moment — like the Dagster integration, which needs a new maintainer. OpenLineage is like any other open standard in that ecosystem coverage is a constant process rather than a journey, and it requires contributions in order to get close to 100%. Thankfully, we are gaining users and contributors all the time, and integrations are being added or improved upon daily. See the Ecosystem page on the website for a list of consumers and producers and links to more resources, and check out the GitHub repo for the codebase, commit history, contributors, governance procedures, and more. We’re quick to respond to messages here and issues on GitHub — usually within one day.
+ + + + diff --git a/html/style.css b/slack-archive/html/style.css similarity index 100% rename from html/style.css rename to slack-archive/html/style.css diff --git a/slack-archive/index.html b/slack-archive/index.html new file mode 100644 index 0000000..a480ca2 --- /dev/null +++ b/slack-archive/index.html @@ -0,0 +1,10 @@ +Slack

Public Channels

Private Channels

    DMs

      Group DMs

        Bots

          Archived Public Channels

            Archived Private Channels

              DMs (Deleted Users)

                \ No newline at end of file diff --git a/search.html b/slack-archive/search.html similarity index 100% rename from search.html rename to slack-archive/search.html