upload file by streamming :shows error log "The operation couldn’t be completed. (kCFErrorDomainCFNetwork error 303.)"

Question

I'm trying to upload big file by streaming, recently I got this error log:

Error Domain=kCFErrorDomainCFNetwork Code=303 "The operation couldn’t be completed. (kCFErrorDomainCFNetwork error 303.)" UserInfo=0x103c0610 {NSErrorFailingURLKey=/adv,/cgi-bin/file_upload-cgic, NSErrorFailingURLStringKey/adv,/cgi-bin/file_upload-cgic}<br>

this is where I set bodystream:

-(void)finishedRequestBody{ // set bodyinput stream
    [self appendBodyString:[NSString stringWithFormat:@"\r\n--%@--\r\n",[self getBoundaryStr]]];
    [bodyFileOutputStream close];
    bodyFileOutputStream = nil;
    //calculate content length
    NSError *fileReadError = nil;
    NSDictionary *fileAttrs = [[NSFileManager defaultManager] attributesOfItemAtPath:pathToBodyFile error:&fileReadError];
    NSAssert1((fileAttrs != nil),@"Couldn't read post body file",fileReadError);
    NSNumber *contentLength = [fileAttrs objectForKey:NSFileSize];

   NSInputStream *bodyStream = [[NSInputStream alloc] initWithFileAtPath:pathToBodyFile];
    [request setHTTPBodyStream:bodyStream];
    [bodyStream release];

    if (staticUpConneciton == nil) {          
        NSURLResponse *response = nil;
        NSError *error = nil;
        NSData *responseData = [NSURLConnection sendSynchronousRequest:request returningResponse:&response error:&error];    
        staticUpConneciton = [[[NSURLConnection alloc]initWithRequest:request delegate:self] retain];                   
    }else{
        staticUpConneciton = [[NSURLConnection connectionWithRequest:request delegate:self]retain];
    }  
}

this is how I write the steam:

    -(void)stream:(NSStream *)stream handleEvent:(NSStreamEvent)eventCode{
        uint8_t buf[1024*100];
        NSUInteger len = 0;
        switch (eventCode) {
            case NSStreamEventOpenCompleted:
                NSLog(@"media file opened");
                break;
            case NSStreamEventHasBytesAvailable:
              //  NSLog(@"should never happened for output stream");
                len = [self.uploadFileInputStream read:buf maxLength:1024];
                if (len) {
                    [self.bodyFileOutputStream write:buf maxLength:len];
                }else{
                    NSLog(@"buf finished wrote %@",self.pathToBodyFile);
                    [self handleStreamCompletion];
                }
                break;
            case NSStreamEventErrorOccurred:
                NSLog(@"stream error");
                break;
            case NSStreamEventEndEncountered:
                NSLog(@"should never for output stream");
                break;
            default:
                break;
        }
}

close stream

-(void)finishMediaInputStream{
    [self.uploadFileInputStream close];
    [self.uploadFileInputStream removeFromRunLoop:[NSRunLoop currentRunLoop] forMode:NSDefaultRunLoopMode];
    self.uploadFileInputStream = nil;
}

-(void)handleStreamCompletion{
    [self finishMediaInputStream];
    // finish requestbody
    [self finishedRequestBody];
}

and I found that error when I implement this method needNewBodyStream: see the following code:

-(NSInputStream *)connection:(NSURLConnection *)connection needNewBodyStream:(NSURLRequest *)request{
    [NSThread sleepForTimeInterval:2];
    NSInputStream *fileStream = [NSInputStream inputStreamWithFileAtPath:pathToBodyFile];
    if (fileStream == nil) {
        NSLog(@"NSURLConnection was asked to retransmit a new body stream for a request. returning nil!");
    }
    return fileStream;
}

this is where I set the headers and mediaInputStream

-(void)setPostHeaders{
    pathToBodyFile = [[NSString alloc] initWithFormat:@"%@%@",NSTemporaryDirectory(),bodyFileName];
    bodyFileOutputStream = [[NSOutputStream alloc] initToFileAtPath:pathToBodyFile append:YES];
    [bodyFileOutputStream open];

    //set bodysteam
    [self appendBodyString:[NSString stringWithFormat:@"--%@\r\n", [self getBoundaryStr]]];
    [self appendBodyString:[NSString stringWithFormat:@"Content-Disposition: form-data; name=\"%@\"\r\n\r\n", @"target_path"]];
    [self appendBodyString:[NSString stringWithFormat:@"/%@",[NSString stringWithFormat:@"%@/%@/%@",UploaderController.getDestination,APP_UPLOADER,[Functions getDateString]]]];
    [self appendBodyString:[NSString stringWithFormat:@"\r\n--%@\r\n", [self getBoundaryStr]]];
    [self appendBodyString:[NSString stringWithFormat:@"Content-Disposition: form-data; name=\"file_path\"; filename=\"%@\"\r\n", fileName]];
    [self appendBodyString:[NSString stringWithString:@"Content-Type: application/octet-stream\r\n\r\n"]];

    NSString *tempFile = [NSTemporaryDirectory() stringByAppendingPathComponent:@"uploadFile"]; 

    NSError *fileReadError = nil;
    NSDictionary *fileAttrs = [[NSFileManager defaultManager] attributesOfItemAtPath:tempFile error:&fileReadError];
    NSAssert1((fileAttrs != nil),@"Couldn't read post body file",fileReadError);
    NSNumber *contentLength = [fileAttrs objectForKey:NSFileSize];
    [request setValue:[contentLength stringValue] forHTTPHeaderField:@"Content-Length"];    
    NSInputStream *mediaInputStream = [[NSInputStream alloc] initWithFileAtPath:tempFile];
    self.uploadFileInputStream = mediaInputStream;    
    [self.uploadFileInputStream setDelegate:self];
    [self.uploadFileInputStream scheduleInRunLoop:[NSRunLoop currentRunLoop] forMode:NSDefaultRunLoopMode];
    [self.uploadFileInputStream open];    
}

this is how I copy data from camera roll

-(void)copyFileFromCamaroll:(ALAssetRepresentation *)rep{
    //copy the file from the camarall to tmp folder (automatically cleaned out every 3 days)
    NSUInteger chunkSize = 100 * 1024;
    NSString *tempFile = [NSTemporaryDirectory() stringByAppendingPathComponent:@"uploadFile"];
    NSLog(@"tmpfile %@",tempFile);
    uint8_t *chunkBuffer = malloc(chunkSize * sizeof(uint8_t));
    NSUInteger length = [rep size];

    NSFileHandle *fileHandle = [[NSFileHandle fileHandleForWritingAtPath: tempFile] retain];
    if(fileHandle == nil) {
        [[NSFileManager defaultManager] createFileAtPath:tempFile contents:nil attributes:nil];
        fileHandle = [[NSFileHandle fileHandleForWritingAtPath:tempFile] retain];
    }

    NSUInteger offset = 0;
    do {
        NSUInteger bytesCopied = [rep getBytes:chunkBuffer fromOffset:offset length:chunkSize error:nil];
        offset += bytesCopied;
        NSData *data = [[NSData alloc] initWithBytes:chunkBuffer length:bytesCopied];
        [fileHandle writeData:data];
        [data release];
    } while (offset < length);
    [fileHandle closeFile];
    [fileHandle release];
    free(chunkBuffer);
    chunkBuffer = NULL;          
    NSError *error;
    NSData *fileData = [NSData dataWithContentsOfFile:tempFile options:NSDataReadingMappedIfSafe error:&error];
    if (!fileData) {
        NSLog(@"Error %@ %@", error, [error description]);
        NSLog(@"%@", tempFile);
        //do what you need with the error
    }            
}

anybody, any ideas? Did I miss something?

Answer 1

Edit:

In order to mention this upfront:

In iOS 7, there is probably an easy solution to upload a large file. Please refer to NSURLSession, NSURLSessionTask, especially:

- (NSURLSessionUploadTask *)uploadTaskWithRequest:(NSURLRequest *)request 
                                         fromFile:(NSURL *)fileURL 
                                completionHandler:(void (^)(NSData *data, NSURLResponse *response, NSError *error))completionHandler;

---

Otherwise,

your code has a number of issues:

• The multipart message is not constructed properly (including the content length).

• You use sendSynchronousRequest and mixing them with delegate methods. Delete the line:

  NSData *responseData = [NSURLConnection sendSynchronousRequest:request  returningResponse:&response error:&error];)    
  

• Assuming you want to upload an asset by creating a temporary file, you can accomplish this (creating a temp file from an asset) much more easily. In fact, you don't need the stream delegate method. An alternative approach avoiding the temp file requires a "bound pair of streams" - and then you need the stream delegate. The latter is more complex, though.

Given your requirements, I would strongly recommend to use NSURLConnection in asynchronous mode implementing the delegates.

Your problem still has enough stuff to be split into three or more questions, thus I will restrict to answer only one issue:

When uploading a file to a server there are a few established approaches how this can be accomplished with HTTP. The suggested way (but not the only way) is to use a POST request with a multipart/form-data media type with a special Disposition.

Lets take a look at your code at the part that is related to uploading the file:

The code you provided seems to have an issue in statement

[self appendBodyString:[NSString stringWithFormat:@"\r\n--%@--\r\n",[self getBoundaryStr]]];

at the beginning of method finishedRequestBody. This looks like the "closing delimiter" of a "multi-part body" which must appear after the last part - but not earlier. So, here is a bug.

Now, let's figure out how to construct a correct multipart/form-data message:

Constructing a multipart message for file upload

We assume you already have the file which you want to upload at path pathToBodyFile represented as a NSInputStream. This is done correctly in statement:

NSInputStream *bodyStream = [[NSInputStream alloc] initWithFileAtPath:pathToBodyFile];

The rules to upload a file via a multipart/form-data message are defined in RFC 1867 "Form-based File Upload in HTML" and a whole bunch of related and dependent RFCs which specify the protocol in great detail (you don't need to read it now, but possibly later).

There was a question recently on SO, where I made an attempt to clarify a multipart media type: NSURLRequest Upload Multiple Files. I would recommend to take a look there, too.

A file upload according RFC 1867 is basically a multipart/form-data message, except that it can use a specialized Disposition where you can specify the original file name in a disposition parameter. The related RFCs are RFC 2388 "Returning Values from Forms: multipart/form-data", and a few dozens more, possibly especially relevant RFC 2047, RFC 6657, RFC 2231 ).

Note: if you have any specific question about any details, it is always recommended to read the related RFCs. (finding the newest and actual ones is a challenge though.)

A multipart/form-data message contains a series of parts. A form-data part consists of some "parameter name" or "label" (represented via a disposition header), other optional headers, and a body.

Each part MUST have a content-disposition header (representing the "parameter name" or "label") whose "value" equals "form-data" and which has a name attribute which specifies a field name (usually but not exclusively referring to a field in a "HTTP form"). For example:

content-disposition: form-data; name="fieldname"

Each part may have an optional Content-Type header. If no one is specified, text/plain is assumed.

After the headers (if any) the body follows.

So, a part may be viewed as a "parameter/value" pair (plus some optional headers).

If the body is a file content, you can specify the original file name in the content-disposition with a filename parameter, e.g.:

content-disposition: form-data; name="image"; filename="image.jpg"

Additionally, you SHOULD set the Content-Type header for this part accordingly, matching the actual file type, e.g.:

Content-Type: image/jpeg

A multipart/form-data message body consists of one or more parts. The parts are separated with a boundary.

(How you set the boundaries, is described in more detail in the given link here on SO NSURLRequest Upload Multiple Files and in the related RFCs.)

---

Example:

Upload a file "image.jpg" whose MIME type is "image/jpeg"

Create a HTTP message with method POST and set the Content-Type header to multipart/form-data specifying a boundary:

Content-type: multipart/form-data, boundary=AaB03x

The "multipart body" of a "multipart/form-data" message consisting of one part looks as follows (note: CRLF are explicitly visible):

\r\n--AaB03x\r\n
Content-Disposition: form-data; name="image"; filename="image.jpg"\r\n
Content-Type: image/jpeg\r\n
\r\n<file-content>--AaB03x--

Now, you need to "translate" this "outline" into Objective-C with the use of NSURLConnection and NSURLRequest, which seems to be straight forward at the first glance. However, there arise a couple of subtle problems:

First:

A multipart message body consists of one or more parts. As you can see, a part itself contains the boundary and headers plus the body. Constructing a part body now becomes complex since a part's body is a stream (your file input stream). The task is now to "merge" a NSData object (boundary and headers) and the file input stream resulting in (some abstract) new input source. This new input source together with other parts (if any) are now required to form again a new input source which is finally a NSInputStream representing the whole multipart body of the multipart/form-data request. This eventual input stream must be set as the HTTPBodyStream property of the NSMutableURLRequest.

I admit, this is a challenge requiring a number of helper classes, and its own Unit Tests!

A simplification using a memory mapped file as representation for the large asset file is probably futile, since you need to form (aka merge) a complete multipart body (one or more parts). This will end up as a NSData object containing headers and file content, ultimately allocated on the heap. This is likely to fail for very large assets (>300MByte).

A solution is to use a bound pair of streams (an Input Stream and an OutputStream connected via a fixed size buffer), where the one end, the Output Stream, is used to write all parts (headers and file content via input stream), and the other end, the Input Stream, is used to "bind" to the HTTPBodyStream property.

A solution for this single problem is worth a new SO question. (There are Samples from Apple which demonstrate this technique).

There are existing solutions which makes it easy to setup a multipart/form-data request provided by third party libs. However, even well known third party libraries do struggle to get this right.

Second:

A Warning:

Like any "language", the HTTP protocol is very picky about the correct syntax, that is - occurrences of separator elements, character encoding, escaping and quoting, etc. For example, if you miss a CRLF or if you miss to apply the proper encoding for a certain string (e.g. a file name) or if you not apply quoting where necessary within some element of the protocol (e.g. boundary or a filename), your server may not understand the message, or misinterpret it.

There is a huge bunch of RFCs that try to unambiguously specify the nitty gritty details. But careful, finding the RFC which actually specifies the current problem will require some effort. And RFCs get updated and obsoleted once in a while ending in a different "current" RFC. So, keep this in mind when writing code: there might be edge cases where your code is not written according the current RFC and you get unexpected behavior.

So, you might now take the challenge - and this is really advanced stuff - and try to implement a "multipart/form-data body as NSInputStream" correctly, or you try a third party solution, which may work under certain conditions, or sometimes not.

---

Tips and Hints for a File Upload with NSURLRequest

• For larger files use a NSInputStream representation for the file, instead of a NSData representation. (You may experiment with mapped file and NSData though, too).

• When setting a NSInputStream as request body, don't open the input stream.

• When setting a NSInputStream as request body you must override the connection:needNewBodyStream: delegate method and provide a new stream object again. (You did this correctly, though I don't understand the purpose of the delay.)

• When providing an input stream as request body without setting a Content-Length header explicitly, NSURLConnection will use "chunked transfer encoding". Usually, this isn't a problem for the server - but in cases where it is, you might set the Content-Length explicitly (if you can determine the length) and NSURLConnection will not use "chunked transfer encoding" to transmit the request body anymore.

• When setting the "Content-Length" header, be sure you set the correct length.

• When using a NSData object as request body, you don't need to set a Content-Length header, NSURLConnection will set this automatically, unless specified explicitly.

• File names in the Disposition header possibly require quoting and encoding (see RFC 2231).

Answer 2

Currently,I found out what cause this error log is wrong content-length.
Originally, I set the content-length is only due to the size of upload file(not including post data).
this is the wrong code in setPostHeaders method:

NSString *tempFile = [NSTemporaryDirectory() stringByAppendingPathComponent:@"uploadFile"]; 
NSError *fileReadError = nil;
NSDictionary *fileAttrs = [[NSFileManager defaultManager] attributesOfItemAtPath:tempFile error:&fileReadError];
NSAssert1((fileAttrs != nil),@"Couldn't read post body file",fileReadError);
NSNumber *contentLength = [fileAttrs objectForKey:NSFileSize];
[request setValue:[contentLength stringValue] forHTTPHeaderField:@"Content-Length"]; 

and I set the size of Content-Length by using pathToBodyFile (this file includes post data)

NSDictionary *fileAttrs = [[NSFileManager defaultManager] attributesOfItemAtPath:pathToBodyFile error:&fileReadError];
NSAssert1((fileAttrs != nil),@"Couldn't read post body file",fileReadError);
NSNumber *contentLength = [fileAttrs objectForKey:NSFileSize];
//NSLog(@"2 body length %@",[contentLength stringValue]);
[request setValue:[contentLength stringValue] forHTTPHeaderField:@"Content-Length"]

and finally, the error log was gone. I don't know why this work. I had thought that the content length is set to the upload file, but actually, the content length is set to the size of file which include post data and upload file