Google App Engine UTF-8/Ascii Codec Issue

Posted: Monday 15 September 2014

So... Recently, I encountered an error using Google's App Engine SDK for Python.

directory>"directory\DeployToAll.bat"
12:45 AM Application: ### appname ###; version: 1
12:45 AM Host: appengine.google.com
12:45 AM
Starting update of app: ### appname ###, version: 1
12:45 AM Getting current resource limits.
Email: ## email ###
Password for ## email ###:
12:45 AM Scanning files on local disk.
Could not guess mimetype for static/game.udk.  Using application/octet-stream.
Could not guess mimetype for static/game.udk.  Using application/octet-stream.
12:45 AM Cloning 127 static files.
12:45 AM Cloning 23 application files.
12:45 AM Uploading 5 files and blobs.
2014-09-15 00:45:18,967 ERROR appcfg.py:2578 An unexpected error occurred. Abort
ing.
Traceback (most recent call last):
  File "C:\Program Files (x86)\Google\google_appengine\google\appengine\tools\ap
pcfg.py", line 2416, in DoUpload
    self._UploadMissingFiles(missing_files, openfunc)
  File "C:\Program Files (x86)\Google\google_appengine\google\appengine\tools\ap
pcfg.py", line 2557, in _UploadMissingFiles
    self.blob_batcher.Flush()
  File "C:\Program Files (x86)\Google\google_appengine\google\appengine\tools\ap
pcfg.py", line 1458, in Flush
    self.SendBatch()
  File "C:\Program Files (x86)\Google\google_appengine\google\appengine\tools\ap
pcfg.py", line 1417, in SendBatch
    payload,
UnicodeDecodeError: 'ascii' codec can't decode byte 0xff in position 0: ordinal
not in range(128)
12:45 AM Rolling back the update.
Traceback (most recent call last):
  File "C:\Program Files (x86)\Google\google_appengine\appcfg.py", line 126, in

    run_file(__file__, globals())
  File "C:\Program Files (x86)\Google\google_appengine\appcfg.py", line 122, in
run_file
    execfile(_PATHS.script_file(script_name), globals_)
  File "C:\Program Files (x86)\Google\google_appengine\google\appengine\tools\ap
pcfg.py", line 5378, in 
    main(sys.argv)
  File "C:\Program Files (x86)\Google\google_appengine\google\appengine\tools\ap
pcfg.py", line 5369, in main
    result = AppCfgApp(argv).Run()
  File "C:\Program Files (x86)\Google\google_appengine\google\appengine\tools\ap
pcfg.py", line 2976, in Run
    self.action(self)
  File "C:\Program Files (x86)\Google\google_appengine\google\appengine\tools\ap
pcfg.py", line 5026, in __call__
    return method()
  File "C:\Program Files (x86)\Google\google_appengine\google\appengine\tools\ap
pcfg.py", line 3789, in Update
    self._UpdateWithParsedAppYaml(appyaml, self.basepath)
  File "C:\Program Files (x86)\Google\google_appengine\google\appengine\tools\ap
pcfg.py", line 3842, in _UpdateWithParsedAppYaml
    self.UpdateVersion(rpcserver, basepath, appyaml, APP_YAML_FILENAME)
  File "C:\Program Files (x86)\Google\google_appengine\google\appengine\tools\ap
pcfg.py", line 3727, in UpdateVersion
    return appversion.DoUpload(paths, openfunc)
  File "C:\Program Files (x86)\Google\google_appengine\google\appengine\tools\ap
pcfg.py", line 2416, in DoUpload
    self._UploadMissingFiles(missing_files, openfunc)
  File "C:\Program Files (x86)\Google\google_appengine\google\appengine\tools\ap
pcfg.py", line 2557, in _UploadMissingFiles
    self.blob_batcher.Flush()
  File "C:\Program Files (x86)\Google\google_appengine\google\appengine\tools\ap
pcfg.py", line 1458, in Flush
    self.SendBatch()
  File "C:\Program Files (x86)\Google\google_appengine\google\appengine\tools\ap
pcfg.py", line 1417, in SendBatch
    payload,
UnicodeDecodeError: 'ascii' codec can't decode byte 0xff in position 0: ordinal
not in range(128)

"'ascii' codec can't decode byte 0xff in position 0: ordinal"

And... Well.. It sucked.

I wasn't able to push anything onto the static site: http://static.extramaster.net/. Where the error would just appear every time I attempted to upload the files onto the server.

Files crucial to my assessment tasks and major projects due were uploaded weeks before the error plagued the site's deployment. and so, there wasn't a primary drive to have a look at fixing the issue.

And, since, of course, I wasn't concerned over such a minor issue as opposed to studying for exams.

Now, the nature of this error is quite vague and seemingly out-of-the blue. Scanning the files with several scripts in Python and Powershell (being on Windows) for any "UTF" files in the application folder came back with nothing; meaning that this issue has nothing to do with having non-ascii files.

Updating to the latest version of the App Engine SDK (1.9.10) proved to be fruitless, as the same error still popped up and again prevented the static site's deployment.

And so, I came up with a small fix...

In "C:\Program Files (x86)\Google\google_appengine\google\appengine\tools\appcfg.py", the offending line is line 1417.

Put in its context, it looks something like this:

      part = '\n'.join(['',
                        'X-Appcfg-File: %s' % urllib.quote(path),
                        'X-Appcfg-Hash: %s' % _Hash(payload),
                        'Content-Type: %s' % mime_type,
                        'Content-Length: %d' % len(payload),
                        'Content-Transfer-Encoding: 8bit',
                        '',
                        payload,
                       ])

This fix is to change "mime_type" to "str(mime_type)"

      part = '\n'.join(['',
                        'X-Appcfg-File: %s' % urllib.quote(path),
                        'X-Appcfg-Hash: %s' % _Hash(payload),
                        'Content-Type: %s' % str(mime_type),
                        'Content-Length: %d' % len(payload),
                        'Content-Transfer-Encoding: 8bit',
                        '',
                        payload,
                       ])

Which then leads to the successful deployment of your site!

directory>"directory\DeployToAll.bat"
12:53 AM Application: ### appname ###; version: 1
12:53 AM Host: appengine.google.com
12:53 AM
Starting update of app: ### appname ###, version: 1
12:53 AM Getting current resource limits.
Email: ## email ###
Password for ## email ###:
12:53 AM Scanning files on local disk.
Could not guess mimetype for static/game.udk.  Using application/octet-stream.
Could not guess mimetype for static/game.udk.  Using application/octet-stream.
12:53 AM Cloning 127 static files.
12:53 AM Cloning 23 application files.
12:53 AM Uploading 4 files and blobs.
12:53 AM Uploaded 4 files and blobs
12:53 AM Compilation starting.
12:53 AM Compilation completed.
12:53 AM Starting deployment.
12:53 AM Checking if deployment succeeded.
12:53 AM Deployment successful.
12:53 AM Checking if updated app version is serving.
12:53 AM Completed update of app: ### appname ###, version: 1
12:53 AM Uploading index definitions.

directory>