API files command is not working properly, traceback to functionLibrary unpack error

[edaniszewski]

Traceback:

  edaniszewski@BCDL52252:~/gfsmaster/GFS$ python client.py files
  10
  Traceback (most recent call last):
    File "client.py", line 10, in <module>
      API.fileList()
    File "/home/edaniszewski/gfsmaster/GFS/API.py", line 458, in fileList
      data = fL.recv(m)
    File "/home/edaniszewski/gfsmaster/GFS/functionLibrary.py", line 136, in recv
      msgLen = packer.unpack(connection.recv(4))[0]
  struct.error: unpack requires a string argument of length 4

Other commands I have tried seem to work, but for whatever reason, its not allowing me to execute the files command. It looks like it originates in the functionLibrary recv function, something to do with the structs?

[edaniszewski]

I'm not sure what is up with this issue, but after restarting a few components, it seems to work now.. definitely a Heisenbug.

[Berrescuda]

This bug has proven difficult to replicate, but the system is working, so I'm obviously moving this to 1.1

[edaniszewski]

Makes sense! In my down time between test cycles, I think I might know how to replicate it, but I haven't tested it out yet since tests are running so any chance of destabilizing the system would suck.

According to Rohail, that error shows up when a 0 or null is sent, so that narrows down the behavior we are looking for a little.

[edaniszewski]

This same error came up a few times when I was running tests also. I haven't had this error show up from an individual command since I raised the issue, but it seems that under heavier loads, it can surface again, so if we want this to be a viable, scalable system, we should track down the origin of this error.

[rohailaltaf]

Can you make sure the filelist command isn't returning duplicates?

-Rohail Altaf

On Thu, Dec 5, 2013 at 7:39 PM, Erick Daniszewski
[email protected] wrote:

This same error came up a few times when I was running tests also. I haven't had this error show up from an individual command since I raised the issue, but it seems that under heavier loads, it can surface again, so if we want this to be a viable, scalable system, we should track down the origin of this error.

Reply to this email directly or view it on GitHub:
#156 (comment)

[edaniszewski]

I don't know if that pertains to this issue, I raised one just now relating to it, and yes, I intend to. I do not want to simply filter out duplicates in the filelist command because while that would return the proper values, the values within the database would still have the duplicates. Instead of filtering out duplicates, I will find how the duplicates get in there and fix that, which should fix that issue ( #160 )