Main site publish functionality #40

changyang-liu · 2021-02-05T05:46:11Z

This pull request enables publishing to the Daily Bruin main site through WordPress REST API from Kerckhoff. Includes:

HTML article generation from AML
Image uploading
Retrieving tags and authors from Daily Bruin

Change the .env file to specify the desired URL and credentials for publishing. Currently set to DB test site.

a better location

now

Currently beautiful soup is used to get headlines from articles

…f/kerckhoff-server into cliu/wordpress-integrations-tests

Before, the type hint for content was dict, even though it should actually be list

Currently only has tests that do not require kerckhoff packages, google drive, or Wordpress POST requests. These other tests will be added later

The return data provides data allowing posts to be deleted in tests

String concatenation creates too much unnecessary copying

- Add logger for kerckhoff module - Replace print statements with log statements - Remove unnecessary print statements

This reverts commit 4d59e92. Cover images should not be optional

- Log level - Don't capture unit test stdout - Add coverage reports to gitignore

DumboOctopus

I forgot to submit this review a long time ago

DumboOctopus · 2021-02-14T05:38:36Z

\

@@ -0,0 +1,168 @@
+import tempfile


Is this file still used?

Yes, it's the python tempfile package used in image uploading

But it's not used in models.py anymore so I'll remove it

DumboOctopus · 2021-02-14T05:39:38Z

docker-compose.yml

+      MYSQL_DATABASE: wordpress
+      MYSQL_USER: wordpress
+      MYSQL_PASSWORD: wordpress
+  wordpress:


Do you use the wordpress docker container and/or do you think it would be ok to use the docker container instead of the fake mainsite

I don't use the wordpress docker container. We added this a long time ago so I don't quite remember what exactly we were planning. I think I should be able to remove it

DumboOctopus · 2021-02-14T05:40:26Z

kerckhoff/packages/\

@@ -0,0 +1,159 @@
+import tempfile


This file might have been created accidently

DumboOctopus · 2021-02-14T05:41:22Z

kerckhoff/packages/exceptions.py

@@ -6,3 +6,9 @@ class GoogleDriveNotConfiguredException(APIException):

    def __init__(self, package):
        self.detail = f"Google Drive is not yet configured for {package.slug}."
+
+class PublishError(APIException):
+    status_code = 400


Hmm I feel like it might be more of a 500 error since the server was unable to fulfill a valid request (400 means request was bad)

I think it's hard to say, because some of them are due to invalid AML, and others are because the server failed to do something. I think I'll add an argument to specify the error code

ohh I see. Yeah in that case I think an argument would be good

DumboOctopus · 2021-02-14T05:44:33Z

kerckhoff/packages/models.py

+            if(file.file_name == "data.aml"):
+                json_data = file.data
+            if("jpg" in file.file_name or "jpeg" in file.file_name
+               or "png" in file.file_name):


would it be ok if we just checked if the file ended with jpg, jpeg or png? (there might be some built in python package that can do that for us). The reason is because this could possibly lead to us uploading files like "png_list.txt" to wordpress' media library and that might fail causing the whole publish operation to fail.

Makes sense. I'll use the python os package for that. I don't remember if I only allow jpg, jpeg and png because DB only uses those file types, or if I simply forgot to include more. Do you know if kerckhoff should be able to support other image extensions as well?

One thing they would want to have in the future is gifs sometimes. However, there some changes we need to make to aws uploading code to support gifs so for now we can leave it out

The old file was probably getting it from models.py, and it broke after I removed it

DumboOctopus

Just the merge conflict but besides that good to go!

DumboOctopus · 2021-05-18T20:47:57Z

kerckhoff/packages/operations/wordpress_utils.py

+                for chunk in res.iter_content(chunk_size=1000):
+                    f.write(chunk)
+                headers = self.basic_auth_header
+                headers["Content-Disposition"] = f"form-data; filename={file_name}"


This is a small thing but technically modifying headers will modify self.basic_auth_header.

changyang-liu and others added 30 commits January 15, 2020 21:24

Added wordpress and mysql docker containers

9f9df6d

Added a sample program for posting to WordPress API

ff76b84

Updated wordpress api test for application passwords. Also moved to

6939655

a better location

Wordpress integration for publishing packages

37bbfea

Wordpress integration done for html posting

842b146

Refactored wordpress integration into a class. Supports image uploading

e9b6876

now

In paragraph photo for article publishing

68f1530

Added caption uploading for images

a9be5f8

Added suport for saving wordpress author and category ids in models

f892952

Not quite working yet

35b00ed

wasn't running for creating packet set errors

7fdf0ea

Sanitize embed links in AML

eacf086

Add beautiful soup as dependency

84ee730

Currently beautiful soup is used to get headlines from articles

Add html generation for related links

998d45a

Merge branch 'neil/incomplete-testing' of https://github.com/kerckhof…

0b9bd17

…f/kerckhoff-server into cliu/wordpress-integrations-tests

Change type hints for aml content

1e7669d

Before, the type hint for content was dict, even though it should actually be list

Add unit tests for WordpressIntegration

baae3e4

Currently only has tests that do not require kerckhoff packages, google drive, or Wordpress POST requests. These other tests will be added later

Add author and category data for testing

c1e8790

Add image uploading test

e54c1ea

Add Wordpress publishing test

4ca039e

Add return value for publish method

8da1bad

The return data provides data allowing posts to be deleted in tests

Remove unused tests

5af64df

Remove unused imports

c63b7b6

Tests for html generation error checking

5cc5eaa

Fix error message for invalid aml item

5d20532

Make coveimg optional

4d59e92

Fix html generation to be more efficient

e84450b

String concatenation creates too much unnecessary copying

Check for missing image files

390985a

Add uncategorized test

54e3522

Logging fixes and additional logs

9a57cf9

- Add logger for kerckhoff module - Replace print statements with log statements - Remove unnecessary print statements

changyang-liu added 8 commits October 29, 2020 14:32

Add kerckhoff module to loggers

3e62b79

Add published article tests for AML paragraphs

690cd34

Bypass docker database authentication

ed2a6e2

Add html tests for embed items

21b8bf4

Minor cleanup

8a91dab

Revert "Make coveimg optional"

a5e9426

This reverts commit 4d59e92. Cover images should not be optional

Config changes

ecec90f

- Log level - Don't capture unit test stdout - Add coverage reports to gitignore

Merge branch 'master' into cliu/wordpress

80ee2c2

changyang-liu changed the title ~~Cliu/wordpress~~ Main site publish functionality Feb 5, 2021

changyang-liu requested a review from DumboOctopus February 7, 2021 07:48

DumboOctopus reviewed May 14, 2021

View reviewed changes

changyang-liu added 4 commits May 17, 2021 20:33

Remove unused imports

d9ab58a

More robust file extension checking

eaefa5e

Re-add missing base64 import

f1e37ff

The old file was probably getting it from models.py, and it broke after I removed it

Add status code argument for PublishError

378723e

DumboOctopus approved these changes May 25, 2021

View reviewed changes

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Main site publish functionality #40

Main site publish functionality #40

changyang-liu commented Feb 5, 2021

DumboOctopus left a comment

DumboOctopus Feb 14, 2021

changyang-liu May 17, 2021

changyang-liu May 17, 2021

DumboOctopus Feb 14, 2021

changyang-liu May 17, 2021

DumboOctopus Feb 14, 2021

DumboOctopus Feb 14, 2021

changyang-liu May 17, 2021

DumboOctopus May 18, 2021

DumboOctopus Feb 14, 2021

changyang-liu May 17, 2021

DumboOctopus May 18, 2021

DumboOctopus left a comment

DumboOctopus May 18, 2021

Main site publish functionality #40

Are you sure you want to change the base?

Main site publish functionality #40

Conversation

changyang-liu commented Feb 5, 2021

DumboOctopus left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

DumboOctopus left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment