Rewriting file

[Tonkonozhenko]

Hi,

Is it safe to rewrite one file using smart_open with the next code?

source_url = 's3://bla-bla.json.gz'

with smart_open(source_uri, 'r') as source:
    with smart_open(source_uri, 'w') as target:
        for line in source:
            if _should_remove_line(source_key, line):
                target.write(line)

[Tonkonozhenko]

Or this approach is better?

        lines = []
        with smart_open(source_uri, 'r') as source:
            for line in source:
                if not _should_remove_line(source_key, line):
                    lines.append(line)

        with smart_open(target_uri, 'w') as target:
            for line in lines:
                target.write(line)

[mpenkov]

Yeah, this is better, provided you're OK with storing the lines in-memory.

[piskvorky]

Since you're opening the objects in text mode (r, w instead of rb, wb), I'd use an explicit encoding.

Also, a list comprehension will be more Pythonic than that loop.

But yeah, keeping the entire file in RAM feels suboptimal. Especially with smart_open which is a stream-oriented library meant for very large files.

[piskvorky]

I think S3 supports some sort of object versioning, so you could read from one version and while writing to another version of the same object, but it feels a bit dodgy. I'd consider writing to a separate file, and then moving / renaming it when finished, atomically. It probably achieves the same thing more explicitly.

[mpenkov]

@Tonkonozhenko It may be safe:

• Opening an S3 object for writing with smart_open doesn't do actually do anything as far as the remote data is concerned. Writing to that object is what will change things on the S3 side.
• For single-part writes, the remote object will change when you close the file-like object
• For multi-part writes, smart_open will write parts to S3 as necessary, but the resulting object won't be replaced until the multipart upload is completed, again, at closing

... but all the above are implementation details, so I wouldn't count on it working. The other problem is that it may work for some URLs, like S3, but the moment you change that URL to e.g. a local file, your implementation is guaranteed to break.

[piskvorky]

I actually think it would work for local files, at least on Linux. The two nested opens would be two separate file descriptors. But again, in the weeds in terms of implementation details. Atomic copy-and-swap probably safer / more explicit.