Project reinstalling dependencies on wake up?

In my app https://glitch.com/~emoji-gender-reveal, my cron job that hits it has been timing out because waking up the app takes so long. Looking at the logs, the project reinstalls all the dependencies, which adds a huge amount of overhead to the wake up process.

Why is this happening? If the package-lock.json file hasn’t changed since the last run, why does everything have to be reinstalled? Is it something in my project that is misconfigured?

Here are the logs from waking it up a few minutes ago:

Startup logs
node v12.0.0, with pnpm
Installing...
Performing headless installation
Resolving: total 1, reused 0, downloaded 0
Resolving: total 1, reused 0, downloaded 0
Resolving: total 91, reused 90, downloaded 0
Resolving: total 208, reused 207, downloaded 0
Resolving: total 324, reused 323, downloaded 0
Resolving: total 426, reused 425, downloaded 0
Resolving: total 586, reused 585, downloaded 0
Packages: +691
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
Resolving: total 691, reused 691, downloaded 0
Resolving: total 691, reused 691, downloaded 0
Resolving: total 691, reused 691, downloaded 0
Resolving: total 691, reused 691, downloaded 0
Resolving: total 691, reused 691, downloaded 0
Resolving: total 691, reused 691, downloaded 0
Resolving: total 691, reused 691, downloaded 0
Resolving: total 691, reused 691, downloaded 0
Resolving: total 691, reused 691, downloaded 0
Resolving: total 691, reused 691, downloaded 0
Resolving: total 691, reused 691, downloaded 0, done
.../fsevents/1.2.9/node_modules/fsevents install$ node install
.../2.0.0/node_modules/puppeteer install$ node install.js
.../fsevents/1.2.9/node_modules/fsevents install: Done
.../2.0.0/node_modules/puppeteer install: Chromium downloaded to /rbd/pnpm-volume/4590afb2-5992-4b96-886e-0d8110c0224b/node_modules/.registry.npmjs.org/puppeteer/2.0.0/node_modules/puppeteer/.local-chromium/linux-706915
.../2.0.0/node_modules/puppeteer install: Done

dependencies:
+ compression 1.7.4
+ dotenv 8.2.0
+ express 4.17.1
+ is-docker 2.0.0
+ puppeteer 2.0.0
+ twemoji 12.1.4
+ twitter 1.7.1

devDependencies:
+ clean-webpack-plugin 3.0.0
+ css-loader 3.2.0
+ eslint 6.6.0
+ eslint-config-standard 14.1.0
+ eslint-plugin-import 2.18.2
+ eslint-plugin-node 10.0.0
+ eslint-plugin-promise 4.2.1
+ eslint-plugin-standard 4.0.1
+ hard-source-webpack-plugin 0.13.1
+ html-loader 0.5.5
+ html-webpack-plugin 3.2.0
+ style-loader 1.0.0
+ webpack 4.41.2
+ webpack-cli 3.3.10
+ webpack-subresource-integrity 1.3.4

Total install time: 59915ms


> emoji-gender-reveal@1.2.1 build /app
> webpack --mode production

[hardsource:e278d730] Writing new cache e278d730...
[hardsource:e278d730] Tracking node dependencies with: package-lock.json.
Hash: a6edf4689fa27c4ae2bb
Version: webpack 4.41.2
Time: 6783ms
Built at: 12/13/2019 11:32:02 PM

           Asset       Size  Chunks                          Chunk Names
     client.html   1.27 KiB          [emitted]               
       client.js    250 KiB       0  [emitted]        [big]  client
   client.js.map   60.3 KiB       0  [emitted] [dev]         client
  puppeteer.html  488 bytes          [emitted]               
    puppeteer.js    247 KiB       1  [emitted]        [big]  puppeteer
puppeteer.js.map   43.6 KiB       1  [emitted] [dev]         puppeteer
Entrypoint client [big] = client.js client.js.map
Entrypoint puppeteer [big] = puppeteer.js puppeteer.js.map
[0] ./src/emojiFuncs.js 3.56 KiB {0} {1} [built]
[2] ./src/drawFuncs.js + 1 modules 8.73 KiB {0} {1} [built]
    | ./src/drawFuncs.js 7.84 KiB [built]
    | ./src/utils.js 899 bytes [built]
[3] ./src/emoji.json 351 KiB {0} {1} [built]
[4] ./src/client.js 1 KiB {0} [built]
[5] ./src/client.css 618 bytes {0} [built]
[6] /rbd/pnpm-volume/4590afb2-5992-4b96-886e-0d8110c0224b/node_modules/.registry.npmjs.org/css-loader/3.2.0/node_modules/css-loader/dist/cjs.js!./src/client.css 608 bytes {0} [built]
[9] ./src/puppeteer.js 1.26 KiB {1} [built]
    + 3 hidden modules

WARNING in asset size limit: The following asset(s) exceed the recommended size limit (244 KiB).
This can impact web performance.
Assets: 
  client.js (250 KiB)
  puppeteer.js (247 KiB)

WARNING in entrypoint size limit: The following entrypoint(s) combined asset size exceeds the recommended limit (244 KiB). This can impact web performance.
Entrypoints:
  client (250 KiB)
      client.js
  puppeteer (247 KiB)
      puppeteer.js

WARNING in webpack performance recommendations: 
You can limit the size of your bundles by using import() or require.ensure to lazy load some parts of your application.
For more info visit https://webpack.js.org/guides/code-splitting/
Child html-webpack-plugin for "client.html":
     1 asset
    Entrypoint undefined = client.html
    [0] /rbd/pnpm-volume/4590afb2-5992-4b96-886e-0d8110c0224b/node_modules/.registry.npmjs.org/html-webpack-plugin/3.2.0/node_modules/html-webpack-plugin/lib/loader.js!./src/client.html 1.37 KiB {0} [built]
    [2] (webpack)/buildin/global.js 472 bytes {0} [built]
    [3] (webpack)/buildin/module.js 497 bytes {0} [built]
        + 1 hidden module
Child html-webpack-plugin for "puppeteer.html":
     1 asset
    Entrypoint undefined = puppeteer.html
    [0] /rbd/pnpm-volume/4590afb2-5992-4b96-886e-0d8110c0224b/node_modules/.registry.npmjs.org/html-webpack-plugin/3.2.0/node_modules/html-webpack-plugin/lib/loader.js!./src/puppeteer.html 578 bytes {0} [built]
    [2] (webpack)/buildin/global.js 472 bytes {0} [built]
    [3] (webpack)/buildin/module.js 497 bytes {0} [built]
        + 1 hidden module

> emoji-gender-reveal@1.2.1 serve /app
> node ./server.js

🔨🏖 Your app is listening on port 3000

Hi again @noelleleigh,

This might be a question for @Glitch_Response. Maybe it’s an after effect of the of the recent downtime Glitch had. You might wanna wait for @Glitch_Response to come and take a look at this!

1 Like

Have you specified a node version in your package.json?

Yeah, it’s:

"engines": {
    "node": "12"
  }

Which for Glitch, means Node 12.0.0

Could you try enable-npm then run enable-pnpm?

Tried that, and it’s still reinstalling on wake-up.

I’ve had this pnpm install issue on my projects. I think a temporary solution I found is to use Uptime Robot to ping the project url (your one is https://emoji-gender-reveal.glitch.me) every 5 minutes… from there I didn’t get the project reinstalling packages issue as much.

Thanks for your reports @noelleleigh and @3ch!

While it is common for projects to occasionally reinstall dependencies, it should not be happening all the time.

I’ll ask the team to take a look at this.

@3ch can you tell me the name of your project(s) where this is happening?

Also if y’all could let me know the frequency at which this is happening now, compared to what you usually experience, that would be a big help!

2 Likes

Thanks for that @3ch - I have asked our team to look into it.

I just tried sleeping and waking my project 3 times in a row and it reinstalled the dependencies every time.

It should run pnpm as it wakes up, but the dependencies should already be in place, so that should be pretty fast. Can we get a copy of the logs that you’re seeing when it wakes up?

…/0.3.2/node_modules/node-opus install: CC(target) Release/obj.target/libopus/deps/opus/silk/LPC_analysis_filter.o
…/0.3.2/node_modules/node-opus install: CC(target) Release/obj.target/libopus/deps/opus/silk/LPC_inv_pred_gain.o
…/0.3.2/node_modules/node-opus install: CC(target) Release/obj.target/libopus/deps/opus/silk/table_LSF_cos.o
…/0.3.2/node_modules/node-opus install: CC(target) Release/obj.target/libopus/deps/opus/silk/NLSF2A.o
…/0.3.2/node_modules/node-opus install: CC(target) Release/obj.target/libopus/deps/opus/silk/NLSF_stabilize.o
…/0.3.2/node_modules/node-opus install: CC(target) Release/obj.target/libopus/deps/opus/silk/NLSF_VQ_weights_laroia.o
…/0.3.2/node_modules/node-opus install: CC(target) Release/obj.target/libopus/deps/opus/silk/pitch_est_tables.o
…/0.3.2/node_modules/node-opus install: CC(target) Release/obj.target/libopus/deps/opus/silk/resampler.o
…/0.3.2/node_modules/node-opus install: CC(target) Release/obj.target/libopus/deps/opus/silk/resampler_down2_3.o
…/0.3.2/node_modules/node-opus install: CC(target) Release/obj.target/libopus/deps/opus/silk/resampler_down2.o
…/0.3.2/node_modules/node-opus install: CC(target) Release/obj.target/libopus/deps/opus/silk/resampler_private_AR2.o
…/0.3.2/node_modules/node-opus install: CC(target) Release/obj.target/libopus/deps/opus/silk/resampler_private_down_FIR.o
…/0.3.2/node_modules/node-opus install: CC(target) Release/obj.target/libopus/deps/opus/silk/resampler_private_IIR_FIR.o
…/0.3.2/node_modules/node-opus install: CC(target) Release/obj.target/libopus/deps/opus/silk/resampler_private_up2_HQ.o
…/0.3.2/node_modules/node-opus install: CC(target) Release/obj.target/libopus/deps/opus/silk/resampler_rom.o
…/0.3.2/node_modules/node-opus install: CC(target) Release/obj.target/libopus/deps/opus/silk/sigm_Q15.o
…/0.3.2/node_modules/node-opus install: CC(target) Release/obj.target/libopus/deps/opus/silk/sort.o
…/0.3.2/node_modules/node-opus install: CC(target) Release/obj.target/libopus/deps/opus/silk/sum_sqr_shift.o
…/0.3.2/node_modules/node-opus install: CC(target) Release/obj.target/libopus/deps/opus/silk/stereo_decode_pred.o
…/0.3.2/node_modules/node-opus install: CC(target) Release/obj.target/libopus/deps/opus/silk/stereo_encode_pred.o
…/0.3.2/node_modules/node-opus install: CC(target) Release/obj.target/libopus/deps/opus/silk/stereo_find_predictor.o
…/0.3.2/node_modules/node-opus install: CC(target) Release/obj.target/libopus/deps/opus/silk/stereo_quant_pred.o
…/0.3.2/node_modules/node-opus install: CC(target) Release/obj.target/libopus/deps/opus/silk/float/apply_sine_window_FLP.o
…/0.3.2/node_modules/node-opus install: CC(target) Release/obj.target/libopus/deps/opus/silk/float/corrMatrix_FLP.o
…/0.3.2/node_modules/node-opus install: CC(target) Release/obj.target/libopus/deps/opus/silk/float/encode_frame_FLP.o
…/0.3.2/node_modules/node-opus install: CC(target) Release/obj.target/libopus/deps/opus/silk/float/find_LPC_FLP.o
…/0.3.2/node_modules/node-opus install: CC(target) Release/obj.target/libopus/deps/opus/silk/float/find_LTP_FLP.o
…/0.3.2/node_modules/node-opus install: CC(target) Release/obj.target/libopus/deps/opus/silk/float/find_pitch_lags_FLP.o
…/0.3.2/node_modules/node-opus install: CC(target) Release/obj.target/libopus/deps/opus/silk/float/find_pred_coefs_FLP.o
…/0.3.2/node_modules/node-opus install: CC(target) Release/obj.target/libopus/deps/opus/silk/float/LPC_analysis_filter_FLP.o
…/0.3.2/node_modules/node-opus install: CC(target) Release/obj.target/libopus/deps/opus/silk/float/LTP_analysis_filter_FLP.o
…/0.3.2/node_modules/node-opus install: CC(target) Release/obj.target/libopus/deps/opus/silk/float/LTP_scale_ctrl_FLP.o
…/0.3.2/node_modules/node-opus install: CC(target) Release/obj.target/libopus/deps/opus/silk/float/noise_shape_analysis_FLP.o
…/0.3.2/node_modules/node-opus install: CC(target) Release/obj.target/libopus/deps/opus/silk/float/prefilter_FLP.o
…/0.3.2/node_modules/node-opus install: CC(target) Release/obj.target/libopus/deps/opus/silk/float/process_gains_FLP.o
…/0.3.2/node_modules/node-opus install: CC(target) Release/obj.target/libopus/deps/opus/silk/float/regularize_correlations_FLP.o
…/0.3.2/node_modules/node-opus install: CC(target) Release/obj.target/libopus/deps/opus/silk/float/residual_energy_FLP.o
…/0.3.2/node_modules/node-opus install: CC(target) Release/obj.target/libopus/deps/opus/silk/float/solve_LS_FLP.o
…/0.3.2/node_modules/node-opus install: CC(target) Release/obj.target/libopus/deps/opus/silk/float/warped_autocorrelation_FLP.o
…/0.3.2/node_modules/node-opus install: CC(target) Release/obj.target/libopus/deps/opus/silk/float/wrappers_FLP.o
…/0.3.2/node_modules/node-opus install: CC(target) Release/obj.target/libopus/deps/opus/silk/float/autocorrelation_FLP.o
…/0.3.2/node_modules/node-opus install: CC(target) Release/obj.target/libopus/deps/opus/silk/float/burg_modified_FLP.o
…/0.3.2/node_modules/node-opus install: CC(target) Release/obj.target/libopus/deps/opus/silk/float/bwexpander_FLP.o
…/0.3.2/node_modules/node-opus install: CC(target) Release/obj.target/libopus/deps/opus/silk/float/energy_FLP.o
…/0.3.2/node_modules/node-opus install: CC(target) Release/obj.target/libopus/deps/opus/silk/float/inner_product_FLP.o
…/0.3.2/node_modules/node-opus install: CC(target) Release/obj.target/libopus/deps/opus/silk/float/k2a_FLP.o
…/0.3.2/node_modules/node-opus install: CC(target) Release/obj.target/libopus/deps/opus/silk/float/levinsondurbin_FLP.o
…/0.3.2/node_modules/node-opus install: CC(target) Release/obj.target/libopus/deps/opus/silk/float/LPC_inv_pred_gain_FLP.o
…/0.3.2/node_modules/node-opus install: CC(target) Release/obj.target/libopus/deps/opus/silk/float/pitch_analysis_core_FLP.o
…/0.3.2/node_modules/node-opus install: CC(target) Release/obj.target/libopus/deps/opus/silk/float/scale_copy_vector_FLP.o
…/0.3.2/node_modules/node-opus install: CC(target) Release/obj.target/libopus/deps/opus/silk/float/scale_vector_FLP.o
…/0.3.2/node_modules/node-opus install: CC(target) Release/obj.target/libopus/deps/opus/silk/float/schur_FLP.o
…/0.3.2/node_modules/node-opus install: CC(target) Release/obj.target/libopus/deps/opus/silk/float/sort_FLP.o
…/0.3.2/node_modules/node-opus install: AR(target) Release/obj.target/deps/opus.a
…/0.3.2/node_modules/node-opus install: COPY Release/opus.a
…/0.3.2/node_modules/node-opus install: CXX(target) Release/obj.target/node-opus/src/node-opus.o
…/0.3.2/node_modules/node-opus install: SOLINK_MODULE(target) Release/obj.target/node-opus.node
…/0.3.2/node_modules/node-opus install: COPY Release/node-opus.node
…/0.3.2/node_modules/node-opus install: make: Leaving directory ‘/rbd/pnpm-volume/f277a04d-7ce9-4a2d-838a-665c1f2f361a/node_modules/.registry.npmjs.org/node-opus/0.3.2/node_modules/node-opus/build’
…/0.3.2/node_modules/node-opus install: gyp info ok
…/0.3.2/node_modules/node-opus install: Done
dependencies:

  • better-sqlite-pool 0.2.2
  • canvas 2.5.0
  • cowsay 1.4.0
  • dblapi.js 2.3.0
  • discord.js 11.5.1
  • enmap 4.8.7
  • express 4.17.1
  • gamedig 2.0.14
  • google-translate-api 2.3.0
  • moment 2.24.0
  • moment-duration-format 2.3.2
  • mongoose 5.6.8
  • ms 2.1.2
  • node-cmd 3.0.0
  • node-gyp 5.0.3
  • node-opus 0.3.2
  • node-superfetch 0.1.9
  • nodemon 1.19.1
  • os-utils 0.0.14
  • platform 1.3.5
  • pretty-ms 5.0.0
  • quick.db 7.0.0-b22
  • quick.hook 1.5.1
  • rebuild 0.1.2
  • simple-youtube-api 5.2.1
  • snekfetch 4.0.4
  • superagent 5.1.0
  • superfetch 1.1.1
  • twitter 1.7.1
  • weather-js 2.0.0
  • windows-cpu 1.1.0
  • ytdl-core 0.29.5
    Total install time: 280286ms

this it what i can see logged, though it differs per project because of different packages being installed

See my first post for an install log. I guess it actually reuses the deps, but it still takes quite a long time.

I am seeing this as well on my projects. Even projects that are hit by uptimerobot.com every 5min. I have been getting spammed by the “app is down” notifications I have configured on uptimerobot since approximately the morning (PST) of Wed Dec 11,

Example project: HMDWATest

We think this is fixed now. Let us know if you see it again.

2 Likes

Yeah! Mine is no longer installing every time! Thanks!

I’m getting this error since it decided to reinstall dependencies which it has never done before.

Hi @SwimmingWhale

Is this still happening within your project?

1 Like

About 18-20 hours of the day, yes.

Hii DapperManatee64!

I don’t think this is the same problem as the one that various people were experiencing earlier in the thread - for example, your logs do not show the project reinstalling over and over again - the error is that your project is looking for a .node file that it is not finding. Normally making any change, even whitespace, to package.json will cause a reinstall. I notice you have a watch.json with a throttle set to 900000. Would you try deleting or watch.json temporarily, then causing a reinstall, for example by making a whitespace change to package.json?

Johnicholas

1 Like