You signed in with another tab or window.
Reload
to refresh your session.
You signed out in another tab or window.
Reload
to refresh your session.
You switched accounts on another tab or window.
Reload
to refresh your session.
By clicking “Sign up for GitHub”, you agree to our
terms of service
and
privacy statement
. We’ll occasionally send you account related emails.
Already on GitHub?
Sign in
to your account
I was trying out bazel build cache to Google Cloud Storage.
First build which must have populated the cache
https://travis-ci.org/damienpontifex/io/builds/562897963
And then subsequent build that utilised the cache
https://travis-ci.org/damienpontifex/io/builds/562916740
Saw significant reduction of build times to get successful builds across all. This would be great to speed up feedback time on PR changes.
Currently I'm pointing mine to a GCS bucket of my own, but what would it take to support Google owned bucket to assist here? I noticed in
https://www.youtube.com/watch?v=MyuJRUwT5LI
the mention that core TensorFlow is already using some of these distributed components, so that may give insights into hints here?
@damienpontifex
Created a PR
#376
which utilized
--disk_cache
+ Travis CI caching system to help speed up.
Note Travis CI's caching is shared among stages, so I used TRAVIS_JOB_NUMBER's suffix
(12.5 => 5)
to shard the data in order to not mess up with every build job 's configuration.
We have python 3.7 on Ubuntu 16.04 and python 2.7-3.6 for Ubuntu 14.04 and it turns out this could be an issue if cache are shared among 16.04 and 14.04.
Also, Travis CI's cache will override new one if there is a overlap between on job run and another job run so it may not get the latest cache all the time. But still, there is some considerable improvement from my own testing.
We see significant drop in build time (40+min => ~15 min) for each build:
https://travis-ci.org/tensorflow/io/builds/563868019
The only issue, is that macOS python 3.5 and 3.6 build can not complete in 50min so cache was never pushed to Travis CI (and later build can not take cache).
Also, in case there are some cache failure, we may need to clean up Travis CI cache then every build has to start again. We still need to find out a way to limit the non-cache build time to 50min.
