Return an error on failed S3 uploads

The previous change to this file didn't raise the error up to stream_write,
and so the complete_upload function still ran because the loop was only
broken. It errored because the data was already canceled. This is better
than what we had before, which was to silently fail but report success
(even internally to ourselves!) on bad image upload.

This means we discovered a bug where a user could have failed during image
upload, but quay would write that image to the repository, potentially
writing broken images to S3.

endpoints/v1/registry.py

@@ -252,7 +252,11 @@ def put_image_layer(namespace, repository, image_id):
 
   # Stream write the data to storage.
   with database.CloseForLongOperation(app.config):
-    store.stream_write(repo_image.storage.locations, layer_path, sr)
+    try:
+      store.stream_write(repo_image.storage.locations, layer_path, sr)
+    except IOError:
+      logger.exception('Exception when writing image data')
+      abort(520, 'Image %(image_id)s could not be written. Please try again.', image_id=image_id)
 
   # Append the computed checksum.
   csums = []

storage/cloud.py

@@ -167,7 +167,8 @@ class _CloudStorage(BaseStorage):
       except IOError:
         app.metric_queue.put('MultipartUploadFailure', 1)
         mp.cancel_upload()
-        break
+        raise
+
     app.metric_queue.put('MultipartUploadSuccess', 1)
     mp.complete_upload()