keckdaq Observations Logbook

Please add observation logs in reverse chronological order. All data taken for analysis (both observations and instrument tests) should be noted here. Include UT start and stop times (format is YYMMDD hh:mm:ss, as in the log files), and a brief description of the state of the instrument and tests carried out.

keckdaq Run 9 (Rx0, Rx1, Rx2, Rx3, Rx4; Feb 6 2016 - )

LINKS: keckdaq Run 8 | keckdaq Run 7 | keckdaq Run 6 | keckdaq Run 5 | keckdaq Run 4 | keckdaq Run 3 | keckdaq Run 2 | keckdaq Run 1 | keckdaq Run 0 | mapodaq (summer lab tests)

• 191115 21:15:05

  • Blanking plate removed from rx1 21:00 191115.
  • Rx3 compressor off 22:45 UTC
  • Had some trouble retuning but now writing to /data/smurf_data/20191115/1573858477/outputs/1573859000.dat. Steps still there

• 191114 02:14:13

  • Compressor off on rx0 02:13:00. Pulled plug on motor driver 2 minutes later.
  • Heaters all set to 2.0V, 04:24:15
  • Cycle rx13 only, 18:16:19
  • rx0 FPU ~100K. Gauge reads 13 torr.
  • rx2 FPU off at 23:00UTC.

• 191112 19:45:05

  • 191112 02:00 removed forebaffles on rx13. Installed blanking plates on both, without WTF plate
  • Restarted masd on the 4 mce receivers to explicitly zero their biases. We will start pulling crates today.

    191112 19:38:10 mce_stop_acq
    191112 19:41:05 mceGeneric mce_zero_bias, mce=mce0
    191112 19:41:07 mceGeneric mce_zero_bias, mce=mce2
    191112 19:41:08 mceGeneric mce_zero_bias, mce=mce3
    191112 19:41:09 mceGeneric mce_zero_bias, mce=mce4
        

  • MCE power supply off 21:50:04.
  • Removed mce's on rx024. Leaving smurf and rx3 in case we want to do further testing.
  • compressor off on rx4 191113 02:15:16. Pulled plug on motor driver 2 minutes later.
  • 09:43 all pumps, heat switches set to 4.5V to add more heat. FPU above 20K, He3/4 condensers around 50K.
  • Okay maybe that was too much heat. 09:47:01 bumping down to 2.0V each. Much more controlled!
  • 14:50 pressures went up to 1e-2mbar. Heaters bumped up to 4V on pumps, 3V on heatswitches. They're probably not doing very much.
  • 01:48 FPU at 178K, pressures 2.2torr. Pulled rx3 mce out; flipped off the MCE power supply, pulled smurf crate off

• 191111 00:32:48

  • cycleFridge only on rx1 again, so that it is cold for Shawn tomorrow 191111 10:05:36
  • cycleFridge on 4 other receivers, since they're warmed up 191111 19:24:37
  • 191111 19:56:24 Exiting schedule: /data/keckdaq/autoqueue/pending/9_kCMB_50_499.sch
  • 191111 19:56:24 Starting schedule: /data/keckdaq/autoqueue/pending/9_kCMB_50_500.sch
  • Autoqueue immediately picked up the CMB run: 191110 07:35:07 Starting schedule: /data/keckdaq/autoqueue/pending/9_kCMB_50_499.sch
  • rx1 got really unhappy because it was trying to cycle during the mediator crash, got to a bad state where the focal plane temps were floating at 10K and refusing to come down. Tried heating the pumps manually; first time forgot to turn off the heat switches so only got to ~25K, then had to turn everything off and cool 4K plate and He4 condenser back down to drag the temperature down. Tried reheating the pumps and dragged FPU down to ~4.4K. Initiated cycle on rx1:

    191110 07:48:40 cycleFridge enable=0, rx=rx0+rx2+rx3+rx4
    191110 07:49:04 cycleFridge enable=2, rx=rx1
    191110 07:57:22 cycleFridge
    

  • 07:55:52 Aborted CMB schedule to run stare + scan biased on Al (changed the stare and scan schedule config_hw to config_hw_Al), dropped that in autoqueue
  • Installed blanking plate on rx3. Left forebaffle on, so plate just drops in and screws into WTF plate. Around 0800.
  • 191110 08:37:38 Starting schedule: /data/keckdaq/autoqueue/pending/1_stareandscan_al_001.sch
  • rx1 got cold around 13:50UTC. Reenabled masd1, tuned up rx1 through smurf.
  • rx1 cryo adventures made it unhappy; fridge expired after only 4ish hours. Cycled again via keckViewer 20191110 17:43:26.

• 191109 02:20:25

  • Robert is gone :( Marion and Cyndia and Ta-Lee will attempt not to break anything
  • Forebaffles for rx024 removed. Blanking plates installed. rx4 (receiver E) had an Allen ball stuck into its screw, making it impossible to get the forebaffle off. Took the WTF plate and forebaffle off together. Steele will fix it. rx13 forebaffles remain.
  • Fridge cycle completed; rx0 looks good.
  • Taking CMB data with forebaffles neighboring rx3 removed, in case that proves useful for correlated noise debugging
  • Unlocked mount, slewed back to el=65
  • 191109 02:21:36 Starting schedule: /home/keck/sch/mce_autotune.sch
  • 191109 02:26:45 Starting schedule: /data/keckdaq/autoqueue/pending/9_kCMB_50_498.sch
  • 191109 22:32:48 Exiting schedule: /data/keckdaq/autoqueue/pending/9_kCMB_50_498.sch
  • Why was that so quick? Looks like it only did one phase and then gave up. Trying again
  • 191109 23:53:58 Starting schedule: /data/keckdaq/autoqueue/pending/9_kCMB_50_499.sch
  • Realized that masd1 isn't receiving commands. Tried rebooting masd1, to no avail. Abort schedule, reboot all of masd.
  • Trying again: 191109 23:59:20 Starting schedule: /data/keckdaq/autoqueue/pending/9_kCMB_50_499.sch
  • masd1 looks happy again

• 191107 04:49:09 -

  • /home/keck/sch/mce_autotune.sch
  • /data/keckdaq/autoqueue/pending/9_kCMB_50_497.sch
  • rx0 only reached 0.315K, this time I disengaged the connectors on the cryostat side of the HK backpack
  • FPU temperature shot up to 3K and rising
  • looks like a connection between cryostat and HK backpack was the problem
  • rx0 is cooling, did another cycle, reached target temperature
  • 191107 13:42:51 abort_schedule
  • 191107 13:43:11 Starting schedule: /home/keck/sch/mce_autotune.sch
  • 191107 13:48:20 Starting schedule: /data/keckdaq/autoqueue/pending/9_kCMB_50_497.sch
  • 191108 19:44:52 abort_schedule
  • terminated last CMB run for SPUD/Keck :'( :)
  • /home/keck/sch/runend.sch
  • weather: mostly clear, overcast, clearing
  • windows and baffles:

• 191106 23:16:30

  • cyclefridge enable=2, on all rx

• 191105 04:57:01 - 191106 22:51:15

  • /home/keck/sch/mce_autotune.sch
  • /data/keckdaq/autoqueue/pending/9_kCMB_50_496.sch
  • rx0 still warm, trying to cool it back down
  • 191105 13:50:16 abort_schedule
  • restarted with all 5rx again
  • 191105 13:50:21 Starting schedule: /home/keck/sch/mce_autotune.sch
  • 191105 13:53:44 Starting schedule: /data/keckdaq/autoqueue/pending/9_kCMB_50_496.sch
  • rx0 doesn't cool down all the way about 40mk above traget
  • 191105 23:32 started cycle on rx0
  • restarted system check and setup on smurf per request
  • multiple cycles on rx0 but only reached about 0.32K but each successive cycle was a bit lower
  • removing cables and reinserting them didn't help, also command all heaters off
  • /home/keck/sch/runend.sch
  • /home/keck/sch/heaters_off.sch
  • weather: clear, getting hazy
  • windows and baffles: clear

• 191104 23:11:01

  • cyclefridge enable=2, on all rx

• 191104 10:50:56 - 191104 19:56:04

  • /home/keck/sch/mce_autotune.sch
  • 191104 10:54:33 Starting schedule: /data/keckdaq/autoqueue/pending/1_stareandscan_1_002.sch
  • 191104 15:20:06 Starting schedule: /data/keckdaq/autoqueue/pending/1_stareandscan_1_003.sch
  • 191104 17:37:50 Starting schedule: /data/keckdaq/autoqueue/pending/1_stareandscan_1_004.sch
  • rx0 warmed up fast about 2h after the fridgecycle
  • did only a cycle on rx0 which looked good on the graph, but FPU went totally nuts and warmed up to 20K
  • no indication what was wrong, all other temps looked normal!!!??
  • weather: clear
  • windows and baffles: clear

• 191104 04:40:07

  • cyclefridge enable=2, on all rx
  • installed blanking plate on rx1

• 191102 05:08:49 - 191104 04:40:07

  • /home/keck/sch/mce_autotune.sch
  • /data/keckdaq/autoqueue/pending/9_kCMB_50_495.sch
  • 191103 09:47:25 abort_schedule
  • mostly clear running the special schedules
  • the schedules will start with 1 and 2 so I can queue them for the next few hours
  • 191103 09:56:20 Starting schedule: /data/keckdaq/autoqueue/pending/1_stareandscan_1_001.sch
  • 191103 12:03:24 Starting schedule: /data/keckdaq/autoqueue/pending/1_knoise_responsivity_withazscan_1_001.sch
  • 191103 20:50:49 Starting schedule: /data/keckdaq/autoqueue/pending/2_knoise_responsivity_withazscan_1_002.sc
  • /home/keck/sch/runend.sch
  • weather: thin clouds, overcast, mostly clear, clear
  • windows and baffles: clear

• 191101 23:08:05

  • cyclefridge enable=2, on all rx
  • finally rx0 had a good cycle, also the fastest again
  • for whatever reason, examiner plots are only produced for every second run

• 191031 05:16:41 - 191101 23:08:05

  • /home/keck/sch/mce_autotune.sch
  • /data/keckdaq/autoqueue/pending/9_kCMB_50_494.sch
  • rx0 didn't take the fridgecycle during last run well, problems to get it cold
  • problems with cooling rx0
  • /home/keck/sch/runend.sch
  • weather: clear overhead, clear, thin clouds
  • windows and baffles: clear

• 191030 23:18:48

  • cyclefridge enable=2, on all rx

• 191029 05:33:49 - 191030 23:18:48

  • /home/keck/sch/mce_autotune.sch
  • /data/keckdaq/autoqueue/pending/9_kCMB_50_493.sch
  • updated ephem
  • restarted gcp and masd at skydip
  • 191029 14:26:09 Starting schedule: /home/keck/sch/mce_autotune.sch
  • 191029 14:27:59 Starting schedule: /data/keckdaq/autoqueue/pending/9_kCMB_50_493.sch
  • doesn't look like the update of ephem worked
  • 191029 23:19:42 abort_schedule
  • tried the update again, thanks Justin
  • restarted gcp and masd, this time it worked :)
  • 191029 23:21:17 Starting schedule: /home/keck/sch/mce_autotune.sch
  • 191029 23:23:06 Starting schedule: /data/keckdaq/autoqueue/pending/9_kCMB_50_493.sch
  • /home/keck/sch/runend.sch
  • weather: clear, thin clouds, mostly clear
  • windows and baffles: clear

• 191028 23:26:39

  • cyclefridge enable=2, on all rx
  • rx0 cycle used to be the fastest one, but now slower than rx2,3,4, but we gonna make it for another week ;)
  • KeckViewer froze, keck32 unresponsive, restarted keck31 and 32
  • ephem files need updating, will try today during DSCS

• 191027 05:32:24 - 191028 23:26:39

  • /home/keck/sch/mce_autotune.sch
  • /data/keckdaq/autoqueue/pending/9_kCMB_50_492.sch
  • /home/keck/sch/runend.sch
  • weather: hazy, clear
  • windows and baffles: clear

• 191026 23:34:3

  • cyclefridge enable=2, on all rx

• 191025 05:40:16 - 191026 23:34:3

  • /home/keck/sch/mce_autotune.sch
  • /data/keckdaq/autoqueue/pending/9_kCMB_50_491.sch
  • /home/keck/sch/runend.sch
  • weather: hazy
  • windows and baffles: clear

• 191024 23:37:27

  • cyclefridge enable=2, on all rx

• 191023 04:43:56 - 191024 23:37:27

  • /home/keck/sch/mce_autotune.sch
  • /data/keckdaq/autoqueue/pending/9_kCMB_50_490.sch
  • rx1 took a while for the temperatures to normalize and be ready to cycled
  • cycled rx1
  • /home/keck/sch/runend.sch
  • weather: clear
  • windows and baffles: windows clear, very little snow on the baffles

• 191022 23:50:15

  • cyclefridge enable=2, on all rx
  • massive He leak on B high, rx1 that is why the last cycles were so elongated
  • replace bad line with last spare on the right side

• 191021 05:54:46 - 191022 23:50:15

  • /home/keck/sch/mce_autotune.sch
  • /data/keckdaq/autoqueue/pending/9_kCMB_50_489.sch
  • cycled rx1
  • /home/keck/sch/runend.sch
  • weather: clear overhead, clear
  • windows and baffles: clear

• 191020 23:53:12

  • cyclefridge enable=2, on all rx
  • TCM C on rx0 is on???
  • restarted gcp, masd

• 191019 06:03:52 - 191020 23:53:12

  • /home/keck/sch/mce_autotune.sch
  • /data/keckdaq/autoqueue/pending/9_kCMB_50_488.sch
  • keck 33 unresponsive, had to be rebooted but never started to take data
  • /home/keck/sch/runend.sch
  • weather: hazy, mostly clear overhead
  • windows and baffles: clear

• 191019 00:01:04

  • cyclefridge enable=2, on all rx
  • recharged comp 2, 3, 4

• 191017 06:02:43 - 191019 00:01:04

  • /home/keck/sch/mce_autotune.sch
  • /data/keckdaq/autoqueue/pending/9_kCMB_50_487.sch
  • run crashed, but wired, since there were still a few updates
  • 191017 12:47:14 Starting schedule: /home/keck/sch/mce_autotune.sch
  • 191017 12:49:03 Starting schedule: /data/keckdaq/autoqueue/pending/9_kCMB_50_487.sch
  • cycled rx1, did only single cycles ?????
  • /home/keck/sch/runend.sch
  • weather: overcast, clearing, hazy
  • windows and baffles: windows clear, some snow in and on the baffles

• 191017 00:13:51

  • cyclefridge enable=2, on all rx

• 191015 06:52:46 - 191017 00:13:51

  • /home/keck/sch/mce_autotune.sch
  • /data/keckdaq/autoqueue/pending/9_kCMB_50_486.sch
  • only rx2,3,4 are cold for this run
  • /home/keck/sch/runend.sch
  • weather: clear, wind picking up, clouds moving in
  • windows and baffles: windows clear, very little snow on the baffles

• 191015 00:16:47

  • cyclefridge enable=2, on all rx
  • 191015 01:09:09 Starting schedule: /home/keck/sch/starpoint_pole_keck_summer.sch
  • 191015 02:49:33 Exiting schedule: /home/keck/sch/starpoint_pole_keck_summer.sch
  • rx1 warmed up dramaticall, FPU went to 40K, only after pumping on rx1 did the temperature go back down
  • aborting the cycle also affected the other cycles, rx0 FPU warmed up to 6K
  • run another single cycle on all rx, except rx1

• 191013 06:16:40 - 91015 00:16:47

  • /home/keck/sch/mce_autotune.sch
  • /data/keckdaq/autoqueue/pending/9_kCMB_50_485.sch
  • masd4 running again but masd1 is not
  • 191013 06:19:44 abort_schedule
  • rerun rx1 setup
  • 191013 06:22:59 Starting schedule: /home/keck/sch/mce_autotune.sch
  • 191013 06:24:47 Starting schedule: /data/keckdaq/autoqueue/pending/9_kCMB_50_485.sch
  • masd1 still not running
  • /home/keck/sch/runend.sch
  • weather: overcast, blowing snow, clearing
  • windows and baffles: clear

• 191013 00:29:12

  • cyclefridge enable=2, on all rx

• 191011 06:45:21 - 191013 00:29:12

  • /home/keck/sch/mce_autotune.sch
  • /data/keckdaq/autoqueue/pending/9_kCMB_50_484.sch
  • masd4 stopped, restarting keck37 didn't help
  • masd1 not running
  • /home/keck/sch/runend.sch
  • weather: overcast, blowing snow
  • windows and baffles: windows clear, little snow in the baffles

• 191011 02:26:14

  • cyclefridge enable=2, on all rx

• 191009 06:43:11 - 191011 02:26:14

  • /home/keck/sch/mce_autotune.sch
  • /data/keckdaq/autoqueue/pending/9_kCMB_50_483.sch
  • was notfied today that rx1 is doing test during DSCS, that explains strange behaviour
  • system crashed
  • restarted gcp, masd
  • kicked rx1 out of the loop until I hear back
  • 191009 12:06:34 Starting schedule: /home/keck/sch/mce_autotune.sch
  • 191009 12:12:43 Starting schedule: /data/keckdaq/autoqueue/pending/9_kCMB_50_483.sch
  • cycled rx1
  • but did only a single cycle again
  • also some "ghost" schedules showed in the queue, after the crashed run, but removing them acutually removed runend and fridgecycle schedules from the queue
  • so late fridgecycle start, but good enough for the all rx except rx1
  • /home/keck/sch/runend.sch
  • weather: thin clouds, overcast, blowing snow
  • windows and baffles: windows clear, some snow in and on the baffles

• 191009 00:42:14

  • cyclefridge enable=2, on all rx
  • installed the last 2 az spare lines
  • installed baffle for star pointing telescope
  • power cycled all keck computers, because of wired kviewer behaviour during last run

• 191007 07:17:48 - 191009 00:42:14

  • network problem, had to run out to MAPO to start run, frist scanset missing
  • /home/keck/sch/mce_autotune.sch
  • /data/keckdaq/autoqueue/pending/9_kCMB_50_482.sch
  • kviewer seemed a bit strange, downsample < 9 jumped in and out during scanset/li>
  • clk count and bolos kicking in and out on keck34
  • /home/keck/sch/runend.sch
  • weather: thin clouds
  • windows and baffles: windows clear, very little snow in the baffles

• 191007 00:49:34

  • cyclefridge enable=2, on all rx

• 191005 06:58:54 - 91007 00:49:34

  • /home/keck/sch/mce_autotune.sch
  • /data/keckdaq/autoqueue/pending/9_kCMB_50_481.sch
  • keck35 froze, restarted keck35 and masd2
  • /home/keck/sch/runend.sch
  • weather: overcast, blowing snow, clearing, thin clouds
  • windows and baffles: windows clear, very little snow in and on the baffles

• 191005 00:57:54

  • cyclefridge enable=2, on all rx
  • recharged comp 2, 4
  • leak in A high replaced with spare line, another leak on comp 2
  • swapped to spare line
  • installed one new spare
  • swapped C low to spare line, hard to tell with multiple leaks, where He is coming from

• 191003 07:06:46 - 191005 00:57:54

  • /home/keck/sch/mce_autotune.sch
  • /data/keckdaq/autoqueue/pending/9_kCMB_50_480.sch
  • cycled rx1
  • /home/keck/sch/runend.sch
  • weather: thin clouds, overcast, blowing snow
  • windows and baffles: windows clear, some snow in the baffles

• 191003 01:05:47

  • cyclefridge enable=2, on all rx
  • recharged comp 4, leak in A low??

• 191001 07:14:38 - 191003 01:05:47

  • /home/keck/sch/mce_autotune.sch
  • /data/keckdaq/autoqueue/pending/9_kCMB_50_479.sch
  • /home/keck/sch/runend.sch
  • weather: thin clouds
  • windows and baffles: clear

• 191001 01:12:55

  • cyclefridge enable=2, on all rx
  • recharged comp 3, 4
  • A2 osborn roller dropped, shaft was sheered off, but looks like an old "injury"
  • installed a new osborn roller

• 190929 07:21:10 - 191001 01:12:55

  • /home/keck/sch/mce_autotune.sch
  • /data/keckdaq/autoqueue/pending/9_kCMB_50_478.sch
  • cycled rx1
  • /home/keck/sch/runend.sch
  • weather: little haze, thin clouds
  • windows and baffles: clear

• 190929 01:24:36

  • cyclefridge enable=2, on all rx

• 190927 07:19:29 - 190929 01:24:36

  • /home/keck/sch/mce_autotune.sch
  • /data/keckdaq/autoqueue/pending/9_kCMB_50_477.sch
  • mediator crash
  • 190927 16:54:23 Starting schedule: /home/keck/sch/mce_autotune.sch
  • 190927 16:57:24 Starting schedule: /data/keckdaq/autoqueue/pending/9_kCMB_50_477.sch
  • cycled rx1
  • /home/keck/sch/runend.sch
  • weather: clear, little haze
  • windows and baffles: clear

• 190927 01:28:34

  • cyclefridge enable=2, on all rx

• 190925 07:38:13 - 190927 01:28:34

  • /home/keck/sch/mce_autotune.sch
  • /data/keckdaq/autoqueue/pending/9_kCMB_50_476.sch
  • /home/keck/sch/runend.sch
  • weather: clear
  • windows and baffles: clear

• 190925 01:40:19

  • cyclefridge enable=2, on all rx
  • recharged comp 3, 4

• 190923 07:42:11 - 190925 01:40:19

  • /home/keck/sch/mce_autotune.sch
  • /data/keckdaq/autoqueue/pending/9_kCMB_50_475.sch
  • cycled rx1
  • /home/keck/sch/runend.sch
  • weather: mostly clear, thin clouds
  • windows and baffles: windows clear, very little snow on the baffles

• 190923 01:44:31

  • cyclefridge enable=2, on all rx
  • recharged comp 2

• 190921 07:53:57 - 190923 01:44:31

  • /home/keck/sch/mce_autotune.sch
  • /data/keckdaq/autoqueue/pending/9_kCMB_50_474.sch
  • massive az-diff jump, rehomed, data are good after schedule resume
  • 190922 01:11:31 suspend_schedule
  • 190922 01:23:42 resume_schedule
  • cycled rx1
  • /home/keck/sch/runend.sch
  • weather: thin clouds, overcast, clearing
  • windows and baffles: windows clear, little snow in the baffles

• 190921 01:52:28

  • cyclefridge enable=2, on all rx

• 190919 07:54:16 - 190921 01:52:28

  • /home/keck/sch/mce_autotune.sch
  • /data/keckdaq/autoqueue/pending/9_kCMB_50_473.sch
  • kviewer, plots look strange but pages are being updated??
  • also see the plots on the monitor website, so I assume it's just a kview glitch
  • restarting kview, fixed the problem
  • /home/keck/sch/runend.sch
  • weather: mostly clear, thin clouds
  • windows and baffles: windows clear, some snow in and on the baffles

• 190919 02:00:19

  • cyclefridge enable=2, on all rx
  • removed blanking plate on rx1

• 190917 08:09:41 - 190919 02:00:19

  • /home/keck/sch/mce_autotune.sch
  • /data/keckdaq/autoqueue/pending/9_kCMB_50_472.sch
  • cycled rx1
  • 190918 15:57:16 suspend_schedule
  • one az-diff value, sporadically went just over 30k, the data will be good!!!!, rehomed
  • 190918 15:57:16 suspend_schedule
  • /home/keck/sch/runend.sch
  • weather: blowing snow, overcast, clearing
  • windows and baffles: windows clear, some snow in the baffles

• 190917 02:08:19

  • cyclefridge enable=2, on all rx
  • replaced bad He line with new spare

• 190915 08:12:30 - 190917 02:08:19

  • /home/keck/sch/mce_autotune.sch
  • since weather is good I will run k_noise schedule for rx1
  • 190915 08:15:47 Starting schedule: /data/keckdaq/autoqueue/pending/1_knoise_responsivity_12_003.sch
  • 190915 14:39:53 Starting schedule: /data/keckdaq/autoqueue/pending/9_kCMB_50_471.sch
  • /home/keck/sch/runend.sch
  • weather: clear, very cold, storm moving in, blowing snow, overcast
  • windows and baffles:

• 190915 02:16:04

  • cyclefridge enable=2, on all rx
  • leak E high
  • replaced line with spare
  • installed blanking plate on rx1 for knoise test

• 190913 08:25:24 - 190915 02:16:04

  • /home/keck/sch/mce_autotune.sch
  • /data/keckdaq/autoqueue/pending/9_kCMB_50_470.sch
  • cycled rx1
  • /home/keck/sch/runend.sch
  • weather: thin clouds, clear overhead, getting cold again
  • windows and baffles: windows clear, very little snow on the baffles

• 190913 02:23:51

  • cyclefridge enable=2, on all rx
  • recharged comp 5, possible leak

• 190911 08:33:16 - 190913 02:23:51

  • /home/keck/sch/mce_autotune.sch
  • /data/keckdaq/autoqueue/pending/9_kCMB_50_469.sch
  • cycled rx1
  • /home/keck/sch/runend.sch
  • weather: clearing, thin clouds
  • windows and baffles: clear

• 190911 02:31:48

  • cyclefridge enable=2, on all rx
  • recharged comp 5

• 190909 08:41:08 - 190911 02:31:48

  • /home/keck/sch/mce_autotune.sch
  • /data/keckdaq/autoqueue/pending/9_kCMB_50_468.sch
  • 190909 14:21:11 suspend_schedule
  • az-diff jump, replaced readhead 1A
  • rehomed
  • 190909 14:59:33 resume_schedule
  • /home/keck/sch/runend.sch
  • weather: clear, overcast, blowing snow
  • windows and baffles:

• 190909 02:40:02

  • cyclefridge enable=2, on all rx
  • 190909 03:10:36 Starting schedule: /home/keck/sch/starpoint_pole_keck_winter.sch
  • 190909 07:06:59 Exiting schedule: /home/keck/sch/starpoint_pole_keck_winter.sch

• 190907 08:49:00 - 190909 02:40:02

  • /home/keck/sch/mce_autotune.sch
  • /data/keckdaq/autoqueue/pending/9_kCMB_50_467.sch
  • cycled rx1
  • /home/keck/sch/runend.sch
  • weather: thin clouds, clear
  • windows and baffles:

• 190907 02:47:40

  • cyclefridge enable=2, on all rx
  • recharged comp 2, 3, 4

• 190905 08:56:52 - 190907 02:47:40

  • /home/keck/sch/mce_autotune.sch
  • /data/keckdaq/autoqueue/pending/9_kCMB_50_466.sch
  • 190905 16:17:02 suspend_schedule
  • massive az-diff jump, rehomed, data good after resume schedule
  • 190905 16:24:44 resume_schedule
  • cycled rx1
  • /home/keck/sch/runend.sch
  • weather: overcast, blowing snow
  • windows and baffles: overcast, blowing snow, improving, thin clouds

• 190905 03:21:24

  • cyclefridge enable=2, on all rx

• 190903 09:04:43 - 190905 03:17:01

  • /home/keck/sch/mce_autotune.sch
  • /data/keckdaq/autoqueue/pending/9_kCMB_50_465.sch
  • 190905 02:33:13 abort_schedule
  • because weather is really bad I used the last few minutes for some magnetic testing on rx1
  • /home/keck/sch/runend.sch
  • weather: overcast, blowing snow
  • windows and baffles: windows clear, lots of snow in and on the baffles

• • cyclefridge enable=2, on all rx

• 190901 09:12:35 - 190903 03:03:18

  • /home/keck/sch/mce_autotune.sch
  • /data/keckdaq/autoqueue/pending/9_kCMB_50_464.sch
  • cycled rx1
  • /home/keck/sch/runend.sch
  • weather: clouds moving in, blowing snow, overcast
  • windows and baffles: windows clear, lots of snow in the baffles

• 190901 03:14:43

  • 190903 03:03:49 Starting schedule: /home/keck/sch/mce_autotune.sch
  • before fridgecycle, I did some magnetic testing on rx1
  • see obs log "magnetic testing rx1"
  • cyclefridge enable=2, on all rx

• 190830 09:07:33 - 190901 03:14:43

  • /home/keck/sch/mce_autotune.sch
  • /data/keckdaq/autoqueue/pending/9_kCMB_50_463.sch
  • /home/keck/sch/runend.sch
  • weather: mostly clear overhead
  • windows and baffles: clear

• 190830 03:18:48

  • cyclefridge enable=2, on all rx

• 190828 09:19:17 - 190830 03:18:48

  • /home/keck/sch/mce_autotune.sch
  • /data/keckdaq/autoqueue/pending/9_kCMB_50_462.sch
  • cycled rx1
  • /home/keck/sch/runend.sch
  • weather: thin clouds, clear, hazy
  • windows and baffles:

• 190828 03:26:55

  • cyclefridge enable=2, on all rx

• 190826 09:24:19 - 190828 03:26:55

  • /home/keck/sch/mce_autotune.sch
  • /data/keckdaq/autoqueue/pending/9_kCMB_50_461.sch
  • /home/keck/sch/runend.sch
  • weather: mostly clear, hazy, thin clouds
  • windows and baffles: windows clear, some snow in and on the baffles

• 190826 03:35:30

  • cyclefridge enable=2, on all rx

• 190824 09:31:22 - 190826 03:33:19

  • /home/keck/sch/mce_autotune.sch
  • vibration testing on rx1 and all other rx
  • /data/keckdaq/autoqueue/pending/9_kCMB_50_460.sch
  • mediator crashed right at end of cmb schedule
  • restarted gcp and masd
  • weather: overcast, clearing
  • windows and baffles: windows clear, very little snow in the baffles

• 190824 03:42:28

  • cyclefridge enable=2, on all rx

• 190822 09:51:54 - 190824 03:42:28

  • /home/keck/sch/mce_autotune.sch
  • /data/keckdaq/autoqueue/pending/9_kCMB_50_459.sch
  • /home/keck/sch/runend.sch
  • weather: overcast, mostly hazy
  • windows and baffles: windows clear, some snow in and on the baffles

• 190822 03:50:16

  • cyclefridge enable=2, on all rx

• 190820 09:57:27 - 190822 03:50:16

  • /home/keck/sch/mce_autotune.sch
  • /data/keckdaq/autoqueue/pending/9_kCMB_50_458.sch
  • /home/keck/sch/runend.sch
  • weather: little hazy, overcast
  • windows and baffles: windows clear, quite a bit of snow in and on the baffles

• 190820 03:57:17

  • cyclefridge enable=2, on all rx
  • recharged comp 2

• 190818 10:03:11 - 190820 03:57:17

  • /home/keck/sch/mce_autotune.sch
  • /data/keckdaq/autoqueue/pending/9_kCMB_50_457.sch
  • cycled rx1
  • /home/keck/sch/runend.sch
  • weather: little haze, mostly clear overhead
  • windows and baffles: windows clear, very little snow on the baffles

• 190818 04:19:12

  • cyclefridge enable=2, on all rx

• 190816 09:38:50 - 190818 04:19:12

  • /home/keck/sch/mce_autotune.sch
  • /data/keckdaq/autoqueue/pending/9_kCMB_50_456.sch
  • vibration test rx1, see log file
  • /home/keck/sch/runend.sch
  • weather: clear, getting hazy
  • windows and baffles: clear

• 190816 04:17:34

  • cyclefridge enable=2, on all rx
  • replaced raid controller in keck34
  • dev 1 shows error, but keck34 boots of dev0

• 190814 10:23:11 - 190816 04:17:34

  • /home/keck/sch/mce_autotune.sch
  • /data/keckdaq/autoqueue/pending/9_kCMB_50_455.sch
  • cycled rx1
  • /home/keck/sch/runend.sch
  • weather: mostly clear, clear, cold
  • windows and baffles: clear

• 190814 04:22:05

  • cyclefridge enable=2, on all rx
  • recharged comp 3, 4

• 190812 10:31:13 - 190814 04:22:05

  • /home/keck/sch/mce_autotune.sch
  • /data/keckdaq/autoqueue/pending/9_kCMB_50_454.sch
  • cycled rx1
  • /home/keck/sch/runend.sch
  • weather: very hazy, clearing
  • windows and baffles: windows clear, some snow in and on the baffles

• 190812 04:28:46

  • cyclefridge enable=2, on all rx

• 190810 10:34:22 - 190812 04:28:46

  • /home/keck/sch/mce_autotune.sch
  • /data/keckdaq/autoqueue/pending/9_kCMB_50_453.sch
  • /home/keck/sch/runend.sch
  • weather: clear, getting hazy
  • windows and baffles: clear

• 190810 04:36:39

  • cyclefridge enable=2, on all rx
  • recharged comp 2

• 190808 10:31:21 - 190810 04:36:39

  • /home/keck/sch/mce_autotune.sch
  • /data/keckdaq/autoqueue/pending/9_kCMB_50_452.sch
  • cycled rx1
  • /home/keck/sch/runend.sch
  • weather: clear, wind easing
  • windows and baffles: clear

• 190808 04:44:30

  • cyclefridge enable=2, on all rx
  • removed blanking plate on rx1

• 190806 10:49:20 - 190808 04:44:30

  • /home/keck/sch/mce_autotune.sch
  • /data/keckdaq/autoqueue/pending/9_kCMB_50_451.sch
  • cycled rx1
  • /home/keck/sch/runend.sch
  • weather: mostly clear, blowing snow
  • windows and baffles: clear

• 190806 04:52:21

  • cyclefridge enable=2, on all rx

• 190804 10:28:26 - 190806 04:52:21

  • /home/keck/sch/mce_autotune.sch
  • /data/keckdaq/autoqueue/pending/9_kCMB_50_450.sch
  • keck 35 unresponsive
  • 190804 10:37:52 abort_schedule
  • rebooted keck35 restarted masd
  • 190804 10:42:12 Starting schedule: /home/keck/sch/mce_autotune.sch
  • 190804 10:44:00 Starting schedule: /data/keckdaq/autoqueue/pending/9_kCMB_50_450.sch
  • /home/keck/sch/runend.sch
  • weather: clear, wind picking up
  • windows and baffles: clear

• 190804 05:04:45

  • cyclefridge enable=2, on all rx

• 190802 11:10:33 - 190804 05:04:45

  • /home/keck/sch/mce_autotune.sch
  • /data/keckdaq/autoqueue/pending/9_kCMB_50_449.sch
  • cycled rx1
  • /home/keck/sch/runend.sch
  • weather: clear, cold
  • windows and baffles: clear

• 190802 05:07:56

  • cyclefridge enable=2, on all rx
  • 190802 06:37:05 Starting schedule: /home/keck/sch/starpoint_pole_keck_winter.sch
  • 190802 07:52:04 abort_schedule
  • didn't see suddenly any stars, besides totally clear skies and clear camera
  • after restarting the schedule it worked again, started where I left off in the previous starpoint
  • 190802 07:59:26 Starting schedule: /home/keck/sch/starpoint_pole_keck_winter.sch
  • 190802 09:55:38 Exiting schedule: /home/keck/sch/starpoint_pole_keck_winter.sch

• 190731 11:18:24 - 190802 05:07:56

  • /home/keck/sch/mce_autotune.sch
  • /data/keckdaq/autoqueue/pending/9_kCMB_50_448.sch
  • cycled rx1
  • /home/keck/sch/runend.sch
  • weather: clear, cold
  • windows and baffles: clear

• 190731 05:15:51

  • cyclefridge enable=2, on all rx

• 190729 11:26:16 - 190731 05:15:51

  • /home/keck/sch/mce_autotune.sch
  • /data/keckdaq/autoqueue/pending/9_kCMB_50_447.sch
  • /home/keck/sch/runend.sch
  • weather: mostly clear, clear
  • windows and baffles: clear

• 190729 05:31:44

  • cyclefridge enable=2, on all rx

• 190727 11:30:42 - 190729 05:25:28

  • /home/keck/sch/mce_autotune.sch
  • /data/keckdaq/autoqueue/pending/9_kCMB_50_446.sch
  • cycled rx1
  • /home/keck/sch/runend.sch
  • vibration test on rx1 with 0 bias, see log entries "vibration test"
  • weather: clear, hazy, blowing snow, clearing
  • windows and baffles: windows clear, some snow in and on the baffles

• 190727 05:33:24

  • cyclefridge enable=2, on all rx
  • recharged comp 2, 4

• 190725 11:26:41 - 190727 05:33:24

  • /home/keck/sch/mce_autotune.sch
  • vibration test on rx1, see log file
  • /data/keckdaq/autoqueue/pending/9_kCMB_50_445.sch
  • cycled rx1
  • /home/keck/sch/runend.sch
  • weather: clear
  • windows and baffles: clear

• 190725 05:41:15

  • cyclefridge enable=2, on all rx

• 190723 11:49:52 - 190725 05:41:15

  • /home/keck/sch/mce_autotune.sch
  • /data/keckdaq/autoqueue/pending/9_kCMB_50_444.sch
  • Justin asked to cold reboot computers, did during galaxy scan, since weather is still bad
  • 190724 10:23:17 abort_schedule
  • rebooted keck31-37
  • 190724 11:29:33 Starting schedule: /home/keck/sch/mce_autotune.sch
  • 190724 11:32:34 Starting schedule: /data/keckdaq/autoqueue/pending/9_kCMB_50_444.sch
  • /home/keck/sch/runend.sch
  • weather: high winds, blowing snow, very hazy, wind dying, clearing
  • windows and baffles: windows clear, some snow in the baffles

• 190723 05:49:20

  • cyclefridge enable=2, on all rx

• 190721 11:40:17 - 190723 05:49:20

  • /home/keck/sch/mce_autotune.sch
  • /data/keckdaq/autoqueue/pending/9_kCMB_50_443.sch
  • cycled rx1
  • /home/keck/sch/runend.sch
  • weather: high winds, blowing snow, overcast, hazy
  • windows and baffles: windows clear, lots of snow in and on the baffles

• 190721 05:56:59

  • cyclefridge enable=2, on all rx

• 190719 11:47:25 - 190721 05:56:59

  • /home/keck/sch/mce_autotune.sch
  • 190719 11:51:16 Starting schedule: /data/keckdaq/autoqueue/pending/1_knoise_responsivity_12_002.schx
  • 190719 18:17:09 Starting schedule: /data/keckdaq/autoqueue/pending/9_kCMB_50_442.sch
  • /home/keck/sch/runend.sch
  • weather: clear, getting hazy, high winds, blowing snow
  • windows and baffles: windows clear, lots of snow in and on the baffles

• 190719 06:04:53

  • cyclefridge enable=2, on all rx

• 190717 11:52:46 - 190719 06:04:53

  • /home/keck/sch/mce_autotune.sch
  • /data/keckdaq/autoqueue/pending/9_kCMB_50_441.sch
  • 190717 12:21:19 abort_schedule
  • rehomed
  • 190717 12:21:39 Starting schedule: /data/keckdaq/autoqueue/pending/9_kCMB_50_441.sch
  • /home/keck/sch/runend.sch
  • weather: clear, cold, first time below -100F
  • windows and baffles: clear

• 190717 06:12:42

  • cyclefridge enable=2, on all rx

• 190715 12:17:12 - 190717 06:12:42

  • /home/keck/sch/mce_autotune.sch
  • /data/keckdaq/autoqueue/pending/9_kCMB_50_440.sch
  • cycle on rx1
  • powercycled smurf computer and run setup of crate
  • /home/keck/sch/runend.sch
  • weather: clear
  • windows and baffles: clear

• 190715 06:18:56

  • cyclefridge enable=2, on all rx
  • recharged comp 2

• 190713 12:25:08 - 190715 06:18:56

  • /home/keck/sch/mce_autotune.sch
  • /data/keckdaq/autoqueue/pending/9_kCMB_50_439.sch
  • /home/keck/sch/runend.sch
  • weather: clear
  • windows and baffles: clear

• 190713 06:26:44

  • cyclefridge enable=2, on all rx
  • recharged comp 3

• 190711 12:37:03 - 190713 06:26:44

  • /home/keck/sch/mce_autotune.sch
  • /data/keckdaq/autoqueue/pending/9_kCMB_50_438.sch
  • cycle on rx1
  • /home/keck/sch/runend.sch
  • weather: overcast, clearing, clear
  • windows and baffles: clear

• 190711 06:36:19

  • cyclefridge enable=2, on all rx

• 190709 12:44:54 - 190711 06:36:19

  • /home/keck/sch/mce_autotune.sch
  • /data/keckdaq/autoqueue/pending/9_kCMB_50_437.sch
  • /home/keck/sch/runend.sch
  • weather: clear, overcast, warming up
  • windows and baffles: windows clear, some snow in the baffles

• 190709 06:47:02

  • cyclefridge enable=2, on all rx

• 190707 12:52:46 - 190709 06:47:02

  • /home/keck/sch/mce_autotune.sch
  • /data/keckdaq/autoqueue/pending/9_kCMB_50_436.sch
  • cycle on rx1
  • /home/keck/sch/runend.sch
  • weather: mostly clear, but still blowing snow, wind dying, clear
  • windows and baffles: clear

• 190707 06:52:04

  • cyclefridge enable=2, on all rx

• 190705 13:48:23 - 190707 06:52:04

  • /home/keck/sch/mce_autotune.sch
  • /data/keckdaq/autoqueue/pending/9_kCMB_50_435.sch
  • started one scanset late, see below
  • cycle on rx1
  • /home/keck/sch/runend.sch
  • weather: quite hazy, blowing snow, mostly clear
  • windows and baffles: windows clear, very little snow in and on the baffles

• 190705 08:53:52

  • cyclefridge enable=2, on all rx
  • late fridge cycle
  • recharged comp 2, 4

• 190703 13:04:14 - 190705 08:53:52

  • /home/keck/sch/mce_autotune.sch
  • /data/keckdaq/autoqueue/pending/9_kCMB_50_433.sch
  • ATTENTION: run 434 before run 433
  • cycle on rx1
  • mediator crashed
  • 190704 15:38:02 Starting schedule: /home/keck/sch/mce_autotune.sch
  • 190704 15:41:41 Starting schedule: /data/keckdaq/autoqueue/pending/9_kCMB_50_433.sch
  • didn't realize we were in the 2nd half of run, didn't alter the scheudle before restart
  • 190705 08:48:50 abort_schedule
  • /home/keck/sch/runend.sch
  • weather: clear, getting hazy, wind picking up, blowing snow
  • windows and baffles: windows clear, some snow in the baffles

• 190703 07:07:47

  • cyclefridge enable=2, on all rx
  • 190703 08:10:02 Starting schedule: /home/keck/sch/starpoint_pole_keck_winter.sch
  • 190703 12:19:57 Exiting schedule: /home/keck/sch/starpoint_pole_keck_winter.sch

• 190701 12:50:12 - 190703 07:07:47

  • /home/keck/sch/mce_autotune.sch
  • /data/keckdaq/autoqueue/pending/9_kCMB_50_434.sch
  • ATTENTION: run 434 before run 433
  • /home/keck/sch/runend.sch
  • weather: little hazy, haze, clearing
  • windows and baffles: windows clear, very little snow on baffles

• 190701 07:15:54

  • cyclefridge enable=2, on all rx

• 190629 13:24:13 - 190701 07:15:54

  • /home/keck/sch/mce_autotune.sch
  • /data/keckdaq/autoqueue/pending/9_kCMB_50_432.sch
  • cycle on rx1
  • /home/keck/sch/runend.sch
  • weather: mostly clear, wind picking up, blowing snow
  • windows and baffles: clear

• 190629 07:23:30

  • cyclefridge enable=2, on all rx
  • readjusted readhead 3A

• 190627 13:21:05 - 190629 07:23:30

  • /home/keck/sch/mce_autotune.sch
  • /data/keckdaq/autoqueue/pending/9_kCMB_50_431.sch
  • az-diff jump just over 30k rehomed, was during skydip, so all data are good
  • 190627 23:05:48 resume_schedule
  • el current drifting
  • /home/keck/sch/runend.sch
  • weather: hazy, blowing snow, wind dying, clearing
  • windows and baffles: clear

• 190627 07:31:44

  • cyclefridge enable=2, on all rx

• 190625 13:39:57 - 190627 07:31:44

  • /home/keck/sch/mce_autotune.sch
  • /data/keckdaq/autoqueue/pending/9_kCMB_50_430.sch
  • cycled rx1, first cycle had problems
  • cycled rx1 again, this time it worked
  • /home/keck/sch/runend.sch
  • weather: haze, blowing snow, very hazy
  • windows and baffles: windows, 2" spot of snow in the center, quite a bit of snow in and on the baffles

• 190625 07:39:17

  • cyclefridge enable=2, on all rx

• 190623 13:22:09 - 190625 07:39:17

  • /home/keck/sch/mce_autotune.sch
  • /data/keckdaq/autoqueue/pending/9_kCMB_50_429.sch
  • 190624 13:03:12 keck34 running again from raid
  • /home/keck/sch/runend.sch
  • weather: haze, blowing snow
  • windows and baffles:

• 190623 07:49:51

  • cyclefridge enable=2, on all rx
  • keck34 running at the moment via external USB HDD

• 190621 13:54:29 - 190623 07:49:51

  • /home/keck/sch/mce_autotune.sch
  • /data/keckdaq/autoqueue/pending/9_kCMB_50_428.sch
  • rebuilding raid on keck34
  • /home/keck/sch/runend.sch
  • weather: clear, haze
  • windows and baffles:

• 190621 07:54:59

  • cyclefridge enable=2, on all rx
  • recharged comp 2, 3
  • disk problems on keck34 raid degraded
  • swapped bad disk but, problems with the raid controller

• 190619 14:04:26 - 190621 07:54:59

  • /home/keck/sch/mce_autotune.sch
  • /data/keckdaq/autoqueue/pending/9_kCMB_50_427.sch
  • /home/keck/sch/runend.sch
  • weather: hazy, blowing snow, wind easing, clearing
  • windows and baffles: windows clear, very little snow in and on the baffles

• 190619 08:05:40

  • cyclefridge enable=2, on all rx

• 190617 14:11:24 - 190619 08:05:40

  • /home/keck/sch/mce_autotune.sch
  • /data/keckdaq/autoqueue/pending/9_kCMB_50_426.sch
  • el current drift
  • /home/keck/sch/runend.sch
  • weather: hazy, blowing snow
  • windows and baffles: windows clear, some snow in and on the baffles

• 190617 08:10:48

  • cyclefridge enable=2, on all rx

• 190615 14:19:16 - 190617 08:10:48

  • /home/keck/sch/mce_autotune.sch
  • /data/keckdaq/autoqueue/pending/9_kCMB_50_425.sch
  • cycled fridge on rx1
  • /home/keck/sch/runend.sch
  • weather: hazy, clear, cold, hazy, wind picking up
  • windows and baffles: windows clear, lots of snow in and on the baffles

• 190615 08:18:41

  • cyclefridge enable=2, on all rx
  • recharged comp 4

• 190613 14:27:08 - 190615 08:18:41

  • /home/keck/sch/mce_autotune.sch
  • /data/keckdaq/autoqueue/pending/9_kCMB_50_424.sch
  • /home/keck/sch/runend.sch
  • weather: clear, hazy
  • windows and baffles: windows clear, very little snow on the baffles

• 190613 08:26:36

  • cyclefridge enable=2, on all rx

• 190611 14:35:00 - 190613 08:26:36

  • /home/keck/sch/mce_autotune.sch
  • /data/keckdaq/autoqueue/pending/9_kCMB_50_423.sch
  • 190611 22:50 signalhound disconnected antenna
  • 190612 22:28 signalhound connected antenna
  • /home/keck/sch/runend.sch
  • weather: little haze, clear, cold
  • windows and baffles: clear

• 190611 08:34:24

  • cyclefridge enable=2, on all rx

• 190609 14:42:52 - 190611 08:34:24

  • /home/keck/sch/mce_autotune.sch
  • /data/keckdaq/autoqueue/pending/9_kCMB_50_422.sch
  • keck36 unresponsive, needed reboot
  • 190609 15:01:30 abort_schedule
  • 190609 15:01:36 Starting schedule: /home/keck/sch/mce_autotune.sch
  • 190609 15:05:13 Starting schedule: /data/keckdaq/autoqueue/pending/9_kCMB_50_422.sch
  • /home/keck/sch/runend.sch
  • weather: clear, hazy
  • windows and baffles: windows clear, lots of snow in the baffles

• 190609 08:42:29

  • cyclefridge enable=2, on all rx

• 190607 14:50:43 - 190609 08:42:29

  • /home/keck/sch/mce_autotune.sch
  • /data/keckdaq/autoqueue/pending/9_kCMB_50_421.sch
  • cycled fridge on rx1
  • /home/keck/sch/runend.sch
  • weather: mostly clear, clear
  • windows and baffles: clear

• 190607 08:50:12

  • cyclefridge enable=2, on all rx
  • leak in B-high, rx1
  • replaced with spare line

• 190605 14:58:35 - 190607 08:50:12

  • /home/keck/sch/mce_autotune.sch
  • /data/keckdaq/autoqueue/pending/9_kCMB_50_420.sch
  • cycled fridge on rx1
  • /home/keck/sch/runend.sch
  • weather: very hazy, mostly clear, hazy, clearing
  • windows and baffles: windows clear, quite a bit of snow on the baffles

• 190605 08:58:11

  • cyclefridge enable=2, on all rx
  • recharged comp 2, 3, possible leak in 2

• 190603 15:09:52 - 190605 08:58:11

  • /home/keck/sch/mce_autotune.sch
  • /data/keckdaq/autoqueue/pending/9_kCMB_50_419.sch
  • cycled fridge on rx1
  • /home/keck/sch/runend.sch
  • weather: clear, very hazy
  • windows and baffles: windows clear, some snow in and on the baffles

• 190603 09:08:35

  • cyclefridge enable=2, on all rx
  • 190603 10:25:00 Starting schedule: /home/keck/sch/starpoint_pole_keck_winter.sch
  • 190603 13:58:10 Exiting schedule: /home/keck/sch/starpoint_pole_keck_winter.sch

• 190601 15:14:19 - 190603 09:08:35

  • /home/keck/sch/mce_autotune.sch
  • /data/keckdaq/autoqueue/pending/9_kCMB_50_418.sch
  • mediator crashed, restarted gcp, masd
  • 190601 23:21:31 Starting schedule: /home/keck/sch/mce_autotune.sch
  • 190601 23:27:34 Starting schedule: /data/keckdaq/autoqueue/pending/9_kCMB_50_418.sch
  • /home/keck/sch/runend.sch
  • weather: very hazy, clearing, clear
  • windows and baffles: clear

• 190601 09:13:57

  • cyclefridge enable=2, on all rx
  • recharged comp 2
  • installed blanking plate on rx1

• 190530 15:22:11 - 190601 09:13:57

  • /home/keck/sch/mce_autotune.sch
  • /data/keckdaq/autoqueue/pending/9_kCMB_50_417.sch
  • cycled rx1
  • cycled rx1 a second time
  • /home/keck/sch/runend.sch
  • weather: hazy, very hazy
  • windows and baffles: clear, very little snow in and on the baffles

• 190530 09:21:42

  • cyclefridge enable=2, on all rx

• 190528 15:30:03 - 190530 09:21:42

  • /home/keck/sch/mce_autotune.sch
  • /data/keckdaq/autoqueue/pending/9_kCMB_50_416.sch
  • 190528 22:30:13 az-diff jump, readjusted readhead A1
  • data good again after schedule resume
  • cycled rx1
  • /home/keck/sch/runend.sch
  • weather: mostly clear, hazy
  • windows and baffles: clear

• 190528 09:29:56

  • cyclefridge enable=2, on all rx

• 190526 15:37:54 - 190528 09:29:56

  • /home/keck/sch/mce_autotune.sch
  • /data/keckdaq/autoqueue/pending/9_kCMB_50_415.sch
  • 190527 09:59:05 starting cycle on rx1
  • az-diff getting just over 30k, did a homing
  • several el-current drifts
  • /home/keck/sch/runend.sch
  • weather: clear, very hazy, clearing
  • windows and baffles: very little snow in the baffles

• 190526 09:40:01

  • cyclefridge enable=2, on all rx

• 190524 15:45:46 - 190526 09:40:01

  • /home/keck/sch/mce_autotune.sch
  • /data/keckdaq/autoqueue/pending/9_kCMB_50_414.sch
  • /home/keck/sch/runend.sch
  • weather: clear
  • windows and baffles: clear

• 190524 09:45:37

  • cyclefridge enable=2, on all rx
  • 190524 10:26 to 10:29 used radio in MAPO for signalhound test
  • radio signal shows up in signalhound
  • recharged comp 2

• 190522 15:53:38 - 190524 09:45:37

  • /home/keck/sch/mce_autotune.sch
  • /data/keckdaq/autoqueue/pending/9_kCMB_50_413.sch
  • 190523 21:41:54 started cycle on rx1
  • /home/keck/sch/runend.sch
  • weather: hazy, clearing, clear
  • windows and baffles: windows clear, little snow on the baffle

• 190522 09:53:18

  • cyclefridge enable=2, on all rx

• 190520 16:01:30 - 190522 09:53:18

  • /home/keck/sch/mce_autotune.sch
  • /data/keckdaq/autoqueue/pending/9_kCMB_50_412.sch
  • rx1 stayed cold the whole schedule
  • /home/keck/sch/runend.sch
  • weather: clear, hazy, thin clouds, wind picking up
  • windows and baffles: windows clear, some sown in and on the baffles

• 190520 10:01:21

  • cyclefridge enable=2, on all rx
  • installed new spare line #7, 5 left
  • patching bad lines won't work

• 190518 15:59:23 - 190520 10:01:21

  • /home/keck/sch/mce_autotune.sch
  • /data/keckdaq/autoqueue/pending/9_kCMB_50_411.sch
  • 190518 22:00:53 stopped masd1, double cycle on rx1
  • massive He leak comp4 rx3, replaced line with spare
  • replaced plug on forebaffle heater at variac
  • /home/keck/sch/runend.sch
  • weather: little haze, very hazy, clearing, clear
  • windows and baffles: windows clear, very little snow on the baffles

• 190518 10:11:28

  • cyclefridge enable=2, on all rx
  • recharged comp 4
  • propable leak

• 190516 16:12:22 - 190518 10:11:28

  • /home/keck/sch/mce_autotune.sch
  • /data/keckdaq/autoqueue/pending/9_kCMB_50_410.sch
  • /home/keck/sch/runend.sch
  • weather: hazy, blowing snow, clearing a bit, wind dying
  • windows and baffles: windows clear, some snow on the baffles

• 190516 10:34:57

  • fridgecycles failed to start, since rx1 cycle was still in a hang up state
  • had to abort cycles via keckViewer
  • cyclefridge enable=2, on all rx
  • recharged comp 4
  • had to abort cycle on rx1 after all other rx finished
  • started another cycle via keck31 on rx1

• 190514 16:15:07 - 190516 10:19:19

  • /home/keck/sch/mce_autotune.sch
  • /data/keckdaq/autoqueue/pending/9_kCMB_50_409.sch
  • 190514 22:17:23 masd4 stopped, restarted masd4
  • some problems with rx1, restarted all services
  • tried rx1 cycle via keck31, but fridgecycle failed due to new parameters
  • /home/keck/sch/runend.sch
  • weather: very hazy, hazy, blowing snow
  • windows and baffles: windows clear, some snow on the baffles

• 190514 10:26:49

  • cyclefridge enable=2, on all rx
  • recharged comp 2,3
  • rebooted keck33-37

• 190512 16:22:59 - 190514 10:26:49

  • /home/keck/sch/mce_autotune.sch
  • /data/keckdaq/autoqueue/pending/9_kCMB_50_408.sch
  • rx1 warmed up early
  • masd3 stopped
  • /home/keck/sch/runend.sch
  • weather: clear, little haze, very hazy
  • windows and baffles: windows clear, quite a bit of snow on the baffles

• 190512 10:35:04

  • cyclefridge enable=2, on all rx
  • 190512 12:28:26 reset time on keck31, 32 and omega0, now UTC and GPS in KeckViewer show same time again

• 190510 16:30:51 - 190512 10:35:04

  • /home/keck/sch/mce_autotune.sch
  • /data/keckdaq/autoqueue/pending/9_kCMB_50_407.sch
  • rx1 warming up early
  • gps and computer time are off
  • /home/keck/sch/runend.sch
  • weather: clear
  • windows and baffles: clear

• 190510 10:42:55

  • cyclefridge enable=2, on all rx

• 190508 16:38:42 - 190510 10:42:55

  • /home/keck/sch/mce_autotune.sch
  • /data/keckdaq/autoqueue/pending/9_kCMB_50_406.sch
  • 190510 07:14:46 stopped masd1, rx1 already at 2.6K
  • /home/keck/sch/runend.sch
  • weather: hazy, blowing snow, clearing
  • windows and baffles: windows clear, little snow on the baffles

• 190508 10:50:46

  • cyclefridge enable=2, on all rx

• 190506 16:46:34 - 190508 10:50:46

  • /home/keck/sch/mce_autotune.sch
  • /data/keckdaq/autoqueue/pending/9_kCMB_50_405.sch
  • /home/keck/sch/runend.sch
  • weather: hazy, blowing snow
  • windows and baffles: windows clear, some snow in and on the baffles

• 190506 11:03:02

  • cyclefridge enable=2, on all rx
  • recharged comp 2

• 190504 16:54:26 - 190506 11:03:02

  • /home/keck/sch/mce_autotune.sch
  • /data/keckdaq/autoqueue/pending/9_kCMB_50_404.sch
  • rx1 warming up early
  • /home/keck/sch/runend1.sch
  • weather: hazy, blowing snow
  • windows and baffles: clear

• 190504 11:08:48

  • cyclefridge enable=2, on all rx

• 190502 17:20:18 - 190504 11:08:48

  • /home/keck/sch/mce_autotune.sch
  • /data/keckdaq/autoqueue/pending/9_kCMB_50_403.sch
  • 190502 22:43:54 run died due to work on rx1
  • 190503 03:51:53 Starting schedule: /home/keck/sch/mce_autotune.sch
  • 190503 03:53:42 Starting schedule: /data/keckdaq/autoqueue/pending/9_kCMB_50_403.sch
  • run setup and check on rx1
  • massive He leak in C high, rx2
  • replaced bad line with new one
  • /home/keck/sch/runend1.sch
  • weather: clear, getting hazy, high winds, blowing snow
  • windows and baffles: windows clear, some snow in and on the baffles

• 190502 11:16:27

  • cyclefridge enable=2, on all rx
  • 190502 12:31:57 Starting schedule: /home/keck/sch/starpoint_pole_keck_winter.sch
  • 190502 16:35:22 Exiting schedule: /home/keck/sch/starpoint_pole_keck_winter.sch

• 190430 17:10:10 - 190502 11:16:27

  • /home/keck/sch/mce_autotune.sch
  • /data/keckdaq/autoqueue/pending/9_kCMB_50_402.sch
  • 190502 00:09:40 suspend_schedule, huge az-diff jump, rehomed
  • 190502 00:19:17 resume_schedule
  • /home/keck/sch/runend.sch
  • weather: clear
  • windows and baffles: windows clear, very little snow in and on the baffle

• 190430 11:30:56

  • cyclefridge enable=2, on all rx
  • massive He leak A high, rx0, swapped to spare
  • installed new spare line

• 190428 17:18:01 - 190430 11:30:56

  • /home/keck/sch/mce_autotune.sch
  • /data/keckdaq/autoqueue/pending/9_kCMB_50_401.sch
  • rx1 warmed up quite a bit, more than other times
  • mediator crashed
  • restarted gcp and masd
  • 190430 07:04:40 Starting schedule: /home/keck/sch/mce_autotune.sch
  • 190430 07:06:28 Starting schedule: /data/keckdaq/autoqueue/pending/9_kCMB_50_401.sch
  • /home/keck/sch/runend1.sch
  • weather: hazy, clear
  • windows and baffles: clear

• 190428 11:27:28

  • cyclefridge enable=2, on all rx
  • recharged comp 4

• 190426 17:25:53 - 190428 11:27:28

  • /home/keck/sch/mce_autotune.sch
  • /data/keckdaq/autoqueue/pending/9_kCMB_50_400.sch
  • /home/keck/sch/runend1.sch
  • weather: thin clouds, hazy
  • windows and baffles: windows clear, some snow in and on the baffles

• 190426 11:35:19

  • cyclefridge enable=2, on all rx
  • installed new He spare line, 8 left

• 190424 16:43:11 - 190426 11:35:19

  • starting run without rx1, great conditions so don't want to let the other rx wait
  • /home/keck/sch/mce_autotune.sch
  • /data/keckdaq/autoqueue/pending/9_kCMB_50_399.sch
  • rx1 warmed up early, did an early cycle
  • this cycle looked normal!!! so changes worked in this respect
  • /home/keck/sch/runend.sch
  • weather: clear, thin clouds, warming
  • windows and baffles: clear

• 190424 11:44:46

  • cyclefridge enable=2, on all rx
  • He leak A low, which was anticipated
  • replaced bad line with spare line

• 190422 17:50:08 - 190424 11:44:46

  • /home/keck/sch/mce_autotune.sch
  • /data/keckdaq/autoqueue/pending/9_kCMB_50_398.sch
  • run stopped, since that doesn't happen normally I suspect the stuff they did on rx1 during the DSCS pass
  • 190424 01:05:51 Starting schedule: /home/keck/sch/mce_autotune.sch
  • 190424 01:10:13 Starting schedule: /data/keckdaq/autoqueue/pending/9_kCMB_50_398.sch
  • /home/keck/sch/runend.sch
  • weather: clear
  • windows and baffles: clear

• 190422 11:52:09

  • cyclefridge enable=2, on all rx
  • recharged comp 3
  • put new He spare line in, 9 to go

• 190420 20:13:50 - 190422 11:52:09

  • late start due to rx1
  • /home/keck/sch/mce_autotune.sch
  • /data/keckdaq/autoqueue/pending/9_kCMB_50_397.sch
  • 190421 18:07:13 rx1 warming up
  • early cycle on rx1
  • some frequency changes on power but no outage
  • /home/keck/sch/runend.sch
  • weather: clear
  • windows and baffles: clear

• 190420 11:59:00

  • cyclefridge enable=2, on all rx
  • recharged comp 3,5
  • He leak, replaced E low with spare

• 190418 18:12:18 - 190420 11:59:00

  • /home/keck/sch/mce_autotune.sch
  • /data/keckdaq/autoqueue/pending/9_kCMB_50_396.sch
  • 190419 around 00:00, masd1 stopped
  • Ari asked to restart smurf computer, did that and running through setup
  • masd1 working again
  • mediator crashed
  • restarted gcp and masd
  • 190419 11:45:38 Starting schedule: /home/keck/sch/mce_autotune.sch
  • 190419 11:52:39 Starting schedule: /data/keckdaq/autoqueue/pending/9_kCMB_50_396.sch
  • /home/keck/sch/runend.sch
  • weather: thin clouds, clearing, temperature dropping, haze
  • windows and baffles: windows clear, very little snow in and on the baffles

• 190418 12:06:06

  • cyclefridge enable=2, on all rx
  • recharged comp 2,3,5

• 190416 20:03:59 - 190418 12:06:06

  • rx1 had a slow cycle again, missed first scansets
  • /home/keck/sch/mce_autotune.sch
  • /data/keckdaq/autoqueue/pending/9_kCMB_50_395.sch
  • mediator crashed
  • restarted gcp and masd
  • 190418 00:44:10 Starting schedule: /home/keck/sch/mce_autotune.sch
  • 190418 00:50:19 Starting schedule: /data/keckdaq/autoqueue/pending/9_kCMB_50_395.sch
  • rx1 warmed up early, did an early cycle on rx1
  • /home/keck/sch/runend.sch
  • weather: blowing snow, overcast, wind dying, only thin clouds
  • windows and baffles: clear

• 190416 12:14:42

  • cyclefridge enable=2, on all rx
  • recharged comp 3,5

• 190414 18:28:34 - 190416 12:14:42

  • /home/keck/sch/mce_autotune.sch
  • /data/keckdaq/autoqueue/pending/9_kCMB_50_394.sch
  • /home/keck/sch/runend.sch
  • weather: thin clouds, blowing snow, overcast, haze
  • windows and baffles:

• 190414 12:22:42

  • cyclefridge enable=2, on all rx
  • recharged comp 3,5

• 190412 20:05:39 - 190414 12:22:42

  • missed the first 2 scansets since rx1 cycle was too slow
  • /home/keck/sch/mce_autotune.sch
  • /data/keckdaq/autoqueue/pending/9_kCMB_50_393.sch
  • mediator crashed
  • restarted gcp and masd
  • 190413 02:04:49 Starting schedule: /home/keck/sch/mce_autotune.sch
  • 190413 02:08:26 Starting schedule: /data/keckdaq/autoqueue/pending/9_kCMB_50_393.sch
  • rx1 warming up early
  • did early double cycle on rx1 to get it ready in time for next run
  • /home/keck/sch/runend.sch
  • weather: mostly clear, clear, thin clouds
  • windows and baffles: windows clear, quite a bit of snow in and on the baffles

• 190412 12:30:31

  • cyclefridge enable=2, on all rx
  • recharged comp 3,5

• 190410 18:43:45 - 190412 12:30:31

  • /home/keck/sch/mce_autotune.sch
  • /data/keckdaq/autoqueue/pending/9_kCMB_50_392.sch
  • about 190412 09:10 very brief power outage
  • no warmup
  • /home/keck/sch/runend.sch
  • weather: mostly clear, haze, mostly clear
  • windows and baffles: clear

• 190410 12:38:24

  • cyclefridge enable=2, on all rx
  • recharged comp 3

• 190408 17:28:41 - 190410 12:38:24

  • /home/keck/sch/mce_autotune.sch
  • /data/keckdaq/autoqueue/pending/9_kCMB_50_391.sch
  • about 190409 07:10 a series of brown outs, unstable power for a few minutes
  • 4 compressors where off or in a state where they wouldn't auto power on
  • no massive warm up, should recover, but still aborted run and turned bias etc off since it idles to a quicker recovery
  • all rx warmed up to about 330mk only rx1 was like a rock in the surf not showing any warm up!?
  • 190409 08:06:03 Starting schedule: /home/keck/sch/mce_autotune.sch
  • 190409 08:09:43 Starting schedule: /data/keckdaq/autoqueue/pending/9_kCMB_50_391.sch
  • 190409 16:41 power outage close to 20min
  • 190409 17:57:01 Starting schedule: /home/keck/sch/mce_autotune.sch
  • 190409 18:00:38 Starting schedule: /data/keckdaq/autoqueue/pending/9_kCMB_50_391.sch
  • this time rx1 warmed up and didn't stay cold long
  • 190409 21:49:48 started fridgecycle on rx1
  • /home/keck/sch/runend.sch
  • weather: clear, cold
  • windows and baffles: clear

• 190408 10:22:20

  • cyclefridge enable=2, on all rx
  • recharged comp 3,4,5

• 190406 19:03:10 - 190408 10:22:20

  • /home/keck/sch/mce_autotune.sch
  • /data/keckdaq/autoqueue/pending/9_kCMB_50_390.sch
  • mediator crashed
  • restarted gcp and masd
  • 190407 10:37:03 Starting schedule: /home/keck/sch/mce_autotune.sch
  • 190407 10:40:49 Starting schedule: /data/keckdaq/autoqueue/pending/9_kCMB_50_390.sch
  • mediator crashed again
  • restarted gcp and masd
  • aborted run
  • /home/keck/sch/runend.sch
  • weather: thin clouds, clear
  • windows and baffles: windows clear, very little snow in the baffles

• 190406 12:54:59

  • cyclefridge enable=2, on all rx
  • recharged comp 3,5

• 190404 19:07:21 - 190406 12:54:59

  • /home/keck/sch/mce_autotune.sch
  • /data/keckdaq/autoqueue/pending/9_kCMB_50_389.sch
  • rx1 warm from the beginning
  • started a single cycle on rz1 before the schedule was finished
  • /home/keck/sch/runend1.sch
  • weather: thin clouds, blowing snow
  • windows and baffles: windows clear, quite a bit of snow in and on the baffles

• 190404 10:14:05

  • cyclefridge enable=2, on all rx
  • 190404 12:11:19 Starting schedule: /home/keck/sch/starpoint_pole_keck_summer.sch
  • 190404 13:42:43 Exiting schedule: /home/keck/sch/starpoint_pole_keck_summer.sch

• 190402 19:25:19 - 190404 10:06:01

  • /home/keck/sch/mce_autotune.sch
  • /data/keckdaq/autoqueue/pending/9_kCMB_50_388.sch
  • late start, rx1 wasn't ready
  • mediator crashed
  • restarted gcp and masd
  • antenna1 wouldn't start, rebooted keck31
  • since there were only 2 scansets left decided to cycle and trying starpointing
  • weather: haze, blowing snow
  • windows and baffles:

• 190402 13:10:00

  • cyclefridge enable=2, on all rx
  • recharged comp 2,3

• 190331 19:23:04 - 190402 13:10:00

  • /home/keck/sch/mce_autotune.sch
  • /data/keckdaq/autoqueue/pending/9_kCMB_50_387.sch
  • mediator crashed
  • restarted gcp and masd
  • 190401 11:05:43 Starting schedule: /home/keck/sch/mce_autotune.sch
  • 190401 11:09:21 Starting schedule: /data/keckdaq/autoqueue/pending/9_kCM
  • /home/keck/sch/runend.sch
  • weather: haze, blowing snow
  • windows and baffles: windows clear, little snow on the baffles

• 190331 13:17:47

  • cyclefridge enable=2, on all rx

• 190329 19:30:56 - 190331 13:17:47

  • /home/keck/sch/mce_autotune.sch
  • /data/keckdaq/autoqueue/pending/1_knoise_responsivity_12_001.sch
  • /data/keckdaq/autoqueue/pending/9_kCMB_50_386.sch
  • rx1 stayed cold :)
  • /home/keck/sch/runend.sch
  • weather: mostly clear, thin clouds, haze, blowing snow
  • windows and baffles: windows clear, little snow in and on the baffles

• 190329 13:25:09

  • cyclefridge enable=2, on all rx

• 190327 19:38:48 - 190329 13:25:09

  • /home/keck/sch/mce_autotune.sch
  • /data/keckdaq/autoqueue/pending/9_kCMB_50_385.sch
  • rx1 warmed up early again
  • stopped masd1
  • started an early fridgecycle on rx1 to see if the next cycle is back to normal length
  • 190329 06:37:11 cycleFridge enable=2, rx=rx1
  • /home/keck/sch/runend1.sch
  • weather: mostly clear, haze, thin clouds
  • windows and baffles: clear

• 190327 13:39:04

  • cyclefridge enable=2, on all rx
  • He leak E Hi, comp 5

• 190325 19:46:40 - 190327 13:39:04

  • /home/keck/sch/mce_autotune.sch
  • /data/keckdaq/autoqueue/pending/9_kCMB_50_384.sch
  • 190326 21:30 rx1 starting to warm???
  • temperature set point on rx1 got changed up North
  • /home/keck/sch/runend1.sch
  • mediator crashed
  • restarted gcp and masd
  • 190327 09:19:53 Starting schedule: /data/keckdaq/autoqueue/pending/9_kCMB_50_384.sch
  • weather: clear, thin clouds
  • windows and baffles: clear

• 190325 13:41:21

  • cyclefridge enable=2, on all rx
  • recharged comp 3
  • el current drifting apart

• 190323 19:39:34 - 190325 13:41:21

  • /home/keck/sch/mce_autotune.sch
  • /data/keckdaq/autoqueue/pending/9_kCMB_50_383.sch
  • /home/keck/sch/runend.sch
  • weather: clear overhead, clear
  • windows and baffles: clear

• 190323 12:03:51

  • cyclefridge enable=2, on all rx

• 190323 02:09:01 - 190323 12:03:51

  • /home/keck/sch/mce_autotune.sch
  • /data/keckdaq/autoqueue/pending/1_knoise_responsivity_11_003.sch
  • /home/keck/sch/runend.sch
  • masd4 unresponsive, restarted keck37
  • weather: clear overhead
  • windows and baffles: clear

• 190322 08:38:13 - 190323 02:09:01

  • /home/keck/sch/mce_autotune.sch
  • /data/keckdaq/autoqueue/pending/1_knoise_responsivity_11_001.sch
  • 190322 17:17:06 Exiting schedule: /data/keckdaq/autoqueue/pending/1_knoise_responsivity_11_001.sch
  • 190322 17:17:06 Starting schedule: /data/keckdaq/autoqueue/pending/1_knoise_responsivity_11_002.sch
  • /home/keck/sch/runend.sch
  • weather: mostly clear overhead
  • windows and baffles: clear

• 190322 02:25:59 - 190322 08:29:14

  • /home/keck/sch/mce_autotune.sch
  • /data/keckdaq/autoqueue/pending/1_knoise_responsivity_11_001.sch
  • rx1 is not running
  • 190322 02:49:40 Exiting schedule: /data/keckdaq/autoqueue/pending/1_knoise_responsivity_11_001.sch
  • trobelshooting rx1, still get some error messages
  • will start run now without rx1
  • 190322 08:10:19 Starting schedule: /home/keck/sch/mce_autotune.sch
  • got rx1 running
  • 190322 08:29:14 abort_schedule

• 190321 17:53:28

  • cyclefridge enable=2, on all rx

---

CMB 2019

---

• 190321 07:29:10

  • cyclefridge enable=2, on all rx

• 190320 09:31:54 - 190320 17:58:55

  • /home/keck/sch/mce_autotune.sch
  • /data/keckdaq/autoqueue/pending/7_ffflat_dslmast_33i_023.sch
  • mediator crashed
  • aborted run
  • removed FFF
  • cool down on all rx nominal, pumped on rx1

• 190320 02:59:53

  • cyclefridge enable=2, on all rx

• 190319 19:15:00 - 190320 02:59:53

  • /home/keck/sch/mce_autotune.sch
  • /data/keckdaq/autoqueue/pending/7_ffflat_dslmast_33h_023.sch

• 190319 12:58:34

  • cyclefridge enable=2, on all rx

• 190319 05:14:12 - 190319 12:58:34

  • /home/keck/sch/mce_autotune.sch
  • /data/keckdaq/autoqueue/pending/7_ffflat_dslmast_33e_023.sch

• 190318 22:25:35

  • cyclefridge enable=2, on all rx

• 190318 14:40:52 - 190318 22:25:35

  • /home/keck/sch/mce_autotune.sch
  • /data/keckdaq/autoqueue/pending/7_ffflat_dslmast_33j_022.sch

• 190318 08:40:42

  • cyclefridge enable=2, on all rx

• 190318 00:56:08 - 190318 08:40:42

  • /home/keck/sch/mce_autotune.sch
  • /data/keckdaq/autoqueue/pending/7_ffflat_dslmast_33i_022.sch

• 190317 18:58:40

  • cyclefridge enable=2, on all rx

• 190317 11:12:10 - 190317 18:58:40

  • /home/keck/sch/mce_autotune.sch
  • /data/keckdaq/autoqueue/pending/7_ffflat_dslmast_33h_022.sch

• 190317 05:10:18

  • cyclefridge enable=2, on all rx

• 190316 21:25:26 - 190317 05:10:18

  • /home/keck/sch/mce_autotune.sch
  • /data/keckdaq/autoqueue/pending/7_ffflat_dslmast_33g_022.sch

• 190316 14:44:31

  • cyclefridge enable=2, on all rx

• 190316 07:00:00 - 190316 14:44:31

  • /home/keck/sch/mce_autotune.sch
  • /data/keckdaq/autoqueue/pending/7_ffflat_dslmast_33f_022.sch

• 190316 00:45:17

  • cyclefridge enable=2, on all rx

• 190315 17:00:00 - 190316 00:45:17

  • /home/keck/sch/mce_autotune.sch
  • /data/keckdaq/autoqueue/pending/7_ffflat_dslmast_33e_022.sch

• 190315 09:54:58

  • cyclefridge enable=2, on all rx

• 190315 02:05:59 - 190315 09:54:58

  • /home/keck/sch/mce_autotune.sch
  • /data/keckdaq/autoqueue/pending/7_ffflat_dslmast_33d_022.sch

• 190314 18:30:29

  • cyclefridge enable=2, on all rx
  • rx2 is not responding to mce_stop_acq etc, restarted masd2
  • rx2 doesn't cool down all the way, stuck at 500mK restarted keck35 and gcp/masd, after
  • mceGeneric mce_zero_bias and mce_stop_acq rx2 FPU cooling down to nominal temp

• 190314 10:46:19 - 190314 18:30:29

  • /home/keck/sch/mce_autotune.sch
  • /data/keckdaq/autoqueue/pending/7_ffflat_dslmast_33c_022.sch

• 190314 04:15:07

  • cyclefridge enable=2, on all rx

• 190313 20:30:00 - 190314 04:15:07

  • /home/keck/sch/mce_autotune.sch
  • /data/keckdaq/autoqueue/pending/7_ffflat_dslmast_33b_022.sch

• 190313 13:40:16

  • cyclefridge enable=2, on all rx

• 190313 05:55:42 - 190313 13:40:16

  • /home/keck/sch/mce_autotune.sch
  • /data/keckdaq/autoqueue/pending/7_ffflat_dslmast_33a_022.sch

• 190312 23:44:06

  • cyclefridge enable=2, on all rx

• 190312 16:00:00

  • /home/keck/sch/mce_autotune.sch
  • /data/keckdaq/autoqueue/pending/7_ffflat_dslmast_33j_021.sch

• 190312 09:34:55

  • cyclefridge enable=2, on all rx

• 190312 01:05:44 - 190312 09:27:14

  • /home/keck/sch/mce_autotune.sch
  • /data/keckdaq/autoqueue/pending/7_ffflat_dslmast_33j_021.sch
  • mediator crashed
  • 190312 09:27:14 abort_schedule

• 190311 19:00:36

  • cyclefridge enable=2, on all rx

• 190311 11:15:46 - 190311 19:00:36

  • /home/keck/sch/mce_autotune.sch
  • /data/keckdaq/autoqueue/pending/7_ffflat_dslmast_33g_021.sch

• 190311 05:15:30

  • cyclefridge enable=2, on all rx

• 190310 21:30:00 - 190311 05:15:30

  • /home/keck/sch/mce_autotune.sch
  • /data/keckdaq/autoqueue/pending/7_ffflat_dslmast_33f_021.sch

• 190310 14:50:04

  • cyclefridge enable=2, on all rx

• 190310 07:05:51 - 190310 14:50:04

  • /home/keck/sch/mce_autotune.sch
  • /data/keckdaq/autoqueue/pending/7_ffflat_dslmast_33i_021.sch

• 190310 00:44:24

  • cyclefridge enable=2, on all rx

• 190309 17:00:00 - 190310 00:44:24

  • /home/keck/sch/mce_autotune.sch
  • /data/keckdaq/autoqueue/pending/7_ffflat_dslmast_33h_021.sch

• 190309 10:07:54

  • cyclefridge enable=2, on all rx

• 190309 02:23:46 - 190309 10:07:54

  • /home/keck/sch/mce_autotune.sch
  • /data/keckdaq/autoqueue/pending/7_ffflat_dslmast_33e_021.sch

• 190308 20:17:13

  • cyclefridge enable=2, on all rx

• 190308 12:31:22 - 190308 20:17:13

  • /home/keck/sch/mce_autotune.sch
  • /data/keckdaq/autoqueue/pending/7_ffflat_dslmast_33d_021.sch

• 190308 06:27:35

  • cyclefridge enable=2, on all rx

• 190307 22:41:09 - 190308 06:27:35

  • /home/keck/sch/mce_autotune.sch
  • /data/keckdaq/autoqueue/pending/7_ffflat_dslmast_33c_021.sch
  • keck36 unresponsive, had to be rebooted, restarted masd3

• 190307 13:05:38

  • cyclefridge enable=2, on all rx

• 190307 05:21:43 - 190307 13:05:38

  • /home/keck/sch/mce_autotune.sch
  • /data/keckdaq/autoqueue/pending/7_ffflat_dslmast_33b_021.sch

• 190306 22:58:47

  • cyclefridge enable=2, on all rx

• 190306 12:11:45 - 190306 22:50:16

  • /home/keck/sch/mce_autotune.sch
  • /data/keckdaq/autoqueue/pending/7_ffflat_dslmast_33b_021.sch
  • mediator crashed
  • 190306 22:50:16 abort_schedule

• 190306 05:51:4

  • cyclefridge enable=2, on all rx

• 190305 22:06:19 - 190306 05:51:4

  • moved mirror to "forwardest" position Front bottom (A&B): 10.74" Front top (E&F): 53.84" Back (G&H): 30.47in

  • /home/keck/sch/mce_autotune.sch
  • /data/keckdaq/autoqueue/pending/7_ffflat_dslmast_33a_021.sch
  • el current drifting apart

• 190304 20:11:39

  • cyclefridge enable=2, on all rx
  • recharged comp 3,4,5

• 190304 12:27:24 - 190304 20:11:38

  • /home/keck/sch/mce_autotune.sch
  • /data/keckdaq/autoqueue/pending/7_ffflat_dslmast_33b_020.sch

• 190304 05:35:22

  • cyclefridge enable=2, on all rx

• 190303 21:51:56 - 190304 05:35:22

  • /home/keck/sch/mce_autotune.sch
  • /data/keckdaq/autoqueue/pending/7_ffflat_dslmast_33b_019.sch

• 190303 10:06:51

  • cyclefridge enable=2, on all rx

• 190303 02:21:41 - 190303 10:06:50

  • /home/keck/sch/mce_autotune.sch
  • /data/keckdaq/autoqueue/pending/7_ffflat_dslmast_33b_018.sch

• 190302 19:54:14

  • cyclefridge enable=2, on all rx

• 190302 12:10:08 - 190302 19:54:14

  • /home/keck/sch/mce_autotune.sch
  • /data/keckdaq/autoqueue/pending/7_ffflat_dslmast_33j_016.sch

• 190302 05:57:04

  • cyclefridge enable=2, on all rx
  • recharged comp 2

• 190301 22:13:23 - 190302 05:57:04

  • /home/keck/sch/mce_autotune.sch
  • /data/keckdaq/autoqueue/pending/7_ffflat_dslmast_33g_017.sch

• 190301 12:33:18

  • cyclefridge enable=2, on all rx

• 190301 04:49:34 - 190301 12:33:18

  • /home/keck/sch/mce_autotune.sch
  • /data/keckdaq/autoqueue/pending/7_ffflat_dslmast_33f_017.sch

• 190228 22:44:38

  • cyclefridge enable=2, on all rx

• 190228 15:00:00 - 190228 22:44:38

  • /home/keck/sch/mce_autotune.sch
  • /data/keckdaq/autoqueue/pending/7_ffflat_dslmast_33b_017.sch

• 190228 09:02:18

  • cyclefridge enable=2, on all rx

• 190228 01:18:39 - 190228 09:02:18

  • /home/keck/sch/mce_autotune.sch
  • /data/keckdaq/autoqueue/pending/7_ffflat_dslmast_33i_016.sch

• 190227 19:17:51

  • cyclefridge enable=2, on all rx

• 190227 11:33:54 - 190227 19:17:51

  • /home/keck/sch/mce_autotune.sch
  • /data/keckdaq/autoqueue/pending/7_ffflat_dslmast_33h_016.sch

• 190227 05:11:39

  • cyclefridge enable=2, on all rx

• 190226 21:27:24 - 190227 05:11:39

  • /home/keck/sch/mce_autotune.sch
  • /data/keckdaq/autoqueue/pending/7_ffflat_dslmast_33e_016.sch

• 190226 10:17:24

  • cyclefridge enable=2, on all rx

• 190226 02:31:37 - 190226 10:17:24

  • /home/keck/sch/mce_autotune.sch
  • /data/keckdaq/autoqueue/pending/7_ffflat_dslmast_33d_016.sch

• 190225 20:10:27

  • cyclefridge enable=2, on all rx

• 190225 12:26:33 - 190225 20:10:27

  • /home/keck/sch/mce_autotune.sch
  • /data/keckdaq/autoqueue/pending/7_ffflat_dslmast_33c_016.sch

• 190225 06:23:23

  • cyclefridge enable=2, on all rx

• 190224 22:39:47 - 190225 06:23:23

  • /home/keck/sch/mce_autotune.sch
  • /data/keckdaq/autoqueue/pending/7_ffflat_dslmast_33a_016.sch

• 190224 09:51:26

  • cyclefridge enable=2, on all rx

• 190224 02:07:58 - 190224 09:51:26

  • /home/keck/sch/mce_autotune.sch
  • /data/keckdaq/autoqueue/pending/7_ffflat_dslmast_33g_016.sch

• 190223 19:20:31

  • cyclefridge enable=2, on all rx

• 190223 11:37:28 - 190223 19:20:31

  • /home/keck/sch/mce_autotune.sch
  • /data/keckdaq/autoqueue/pending/7_ffflat_dslmast_33f_016.sch

• 190223 05:09:14

  • cyclefridge enable=2, on all rx

• 190222 21:24:48 - 190223 05:09:14

  • /home/keck/sch/mce_autotune.sch
  • /data/keckdaq/autoqueue/pending/7_ffflat_dslmast_33b_016.sch

• 190222 09:11:34

  • cyclefridge enable=2, on all rx

• 190222 01:28:43 - 190222 09:11:34

  • /home/keck/sch/mce_autotune.sch
  • /data/keckdaq/autoqueue/pending/7_ffflat_dslmast_33j_015.sch
  • masd1 stopped
  • restarted smurf scripts

• 190221 19:53:49

  • cyclefridge enable=2, on all rx

• 190221 12:10:52 - 190221 19:53:49

  • /home/keck/sch/mce_autotune.sch
  • /data/keckdaq/autoqueue/pending/7_ffflat_dslmast_33i_015.sch

• 190221 06:11:32

  • cyclefridge enable=2, on all rx

• 190220 22:29:07 - 190221 06:11:32

  • /home/keck/sch/mce_autotune.sch
  • /data/keckdaq/autoqueue/pending/7_ffflat_dslmast_33h_015.sch

• 190220 14:54:14

  • cyclefridge enable=2, on all rx
  • replaced faulty line from last season with new spare

• 190220 00:22:41 - 190220 14:54:14

  • /home/keck/sch/mce_autotune.sch
  • /data/keckdaq/autoqueue/pending/7_ffflat_dslmast_33e_015.sch
  • masd2 stopped
  • restarted masd2, but since there is no new initialisation, stopped run
  • 190220 02:03:53 abort_schedule
  • rebooted keck33 to 37
  • recharged comp 2 and 3
  • restarted SMuRF services
  • 190220 03:15:13 Starting schedule: /home/keck/sch/mce_autotune.sch
  • masd1 stopped
  • 190220 03:45:26 abort_schedule
  • 190220 04:27:54 Starting schedule: /home/keck/sch/mce_autotune.sch
  • 190220 04:30:12 Starting schedule: /data/keckdaq/autoqueue/pending/7_ffflat_dslmast_33e_015.sch
  • masd1 not updating???
  • 190220 04:31:52 abort_schedule
  • will run this schedule without rx1 until I hear back
  • 190220 04:33:07 Starting schedule: /data/keckdaq/autoqueue/pending/7_ffflat_dslmast_33e_015.sch
  • masd stopped on all rx?
  • 190220 07:13:39 abort_schedule
  • 190220 07:14:20 Starting schedule: /data/keckdaq/autoqueue/pending/7_ffflat_dslmast_33e_015.sch

• 190219 18:12:35

  • cyclefridge enable=2, on all rx

• 190219 10:28:03 - 190219 18:12:35

  • /home/keck/sch/mce_autotune.sch
  • /data/keckdaq/autoqueue/pending/7_ffflat_dslmast_33d_015.sch

• 190219 04:20:20

  • cyclefridge enable=2, on all rx

• 190218 20:38:30 - 190219 04:20:20

  • /home/keck/sch/mce_autotune.sch
  • /data/keckdaq/autoqueue/pending/7_ffflat_dslmast_33c_015.sch

• 190218 13:20:06

  • cyclefridge enable=2, on all rx

• 190218 05:38:04 - 190218 13:20:06

  • /home/keck/sch/mce_autotune.sch
  • /data/keckdaq/autoqueue/pending/7_ffflat_dslmast_33g_015.sch

• 190217 22:55:48

  • cyclefridge enable=2, on all rx

• 190217 15:14:01 - 190217 22:55:48

  • /home/keck/sch/mce_autotune.sch
  • /data/keckdaq/autoqueue/pending/7_ffflat_dslmast_33f_015.sch

• 190217 09:07:03

  • cyclefridge enable=2, on all rx

• 190217 01:24:28 - 190217 09:07:03

  • /home/keck/sch/mce_autotune.sch
  • /data/keckdaq/autoqueue/pending/7_ffflat_dslmast_33b_015.sch

• 190216 19:07:38

  • cyclefridge enable=2, on all rx

• 190216 06:03:36 - 190216 19:07:37

  • /home/keck/sch/mce_autotune.sch
  • SMuRF crate is powered off for autotune
  • 190216 06:05:45 starting fridgecycle on rx 1
  • 190216 06:09:57 Starting schedule: /data/keckdaq/autoqueue/pending/7_ffflat_dslmast_33a_015.sch
  • mediator crashed in a new state?? just restarted mediator
  • 190216 10:28:29 Starting schedule: /data/keckdaq/autoqueue/pending/7_ffflat_dslmast_33a_015.sch
  • mediator crashed again, run was moved to ../done
  • restarted masd and gcp
  • 190216 10:57:26 Starting schedule: /home/keck/sch/mce_autotune.sch
  • 190216 11:25:45 Starting schedule: /data/keckdaq/autoqueue/pending/7_ffflat_dslmast_33a_015.sch
  • 190216 11:27:06 starting one more cycle on rx1

---

Winter 2019

---

• 190215 11:51:29

  • /home/keck/sch/mce_autotune.sch
  • 190215 11:58:58 Starting schedule: /data/keckdaq/autoqueue/pending/7_ffflat_dslmast_33a_015.sch
  • 190215 12:12:54 abort_schedule
  • dk fatal fol error, handcontroller has strange behavior as well, have to investigate
  • handcontroller had some lose screws inside, checked it all, dk still had strange behavior
  • coudln't find anything else wrong, other axis worked ok
  • rebooted keck31, handcontroller works normal again

• 190214 10:05:55

  • cyclefridge enable=2, on rx0,2,3,4

• 190212 07:56:26

  • cyclefridge enable=2, on all rx
  • mediator crashed again in the same wired state as below, just restarting mediator doesn't work
  • keck31 has to be rebooted before control system can be restarted !!??
  • starting another single cycle
  • 190212 11:02:12 Starting schedule: /home/keck/sch/fridgecycle.sch
  • with SMuRF crate powered off the gap between the cycles is narrow again, also the hold time was 30+h
  • mounted the FFF in BACK position, rx1 heating up, after compressores starting again, after 16h FPU about 75K
  • Plug on cryostat is a bit funny, only opened after several tries, then jump in pressure, i.e. vacuum in cryostat
  • not good, pumped for several hours
  • all other rx good for cycle

• 190212 06:13:37

  • cyclefridge enable=2, on all rx
  • smurf crate is powered off for cycle
  • mediator crashed, could not restart until rebooted keck31 and keck32
  • restarted cycle

• 190210 21:21:22

  • cyclefridge enable=2, on all rx

• 190209 20:16:36

  • cyclefridge enable=2, on all rx
  • telescope locked out, pumping on vacuum lines
  • problem with vacuum setup, so we put another pump underneath the rx to bypass the long az wrap line and big roughing pump
  • vacuum on rx1 looked good, pressure change was indifferent when opening cryostat to outside vacuum
  • glycol pump 1 stopped, all compressors turned off, due to overheating, switched to pump 2 manually
  • after cooling down again, switched back to auto mode, pump 1 is running fine again??

• 190209 01:18:56 - 190209 19:20:21

  • 9_kCMB_smurftest_50_015.sch - continued after SMuRF temperature test
  • run died ???
  • 190209 18:49:12 Starting schedule: /data/keckdaq/autoqueue/pending/9_kCMB_smurftest_50_015.sch
  • run died ???
  • 190209 19:20:21 Starting schedule: /data/keckdaq/autoqueue/pending/9_kCMB_smurftest_50_015.sch
  • run died ???
  • aborted run, starting fridgecyle, other stuff to do on the telescope

---

• 190209 06:29

  • Stopped CMB schedule to do SMuRF temperature test with crate off
  • SMuRF crate powered off as per Ari's inastructions
  • After one hour crate is powered back on, and SMuRF is re-initialized.
  • During reinitilization SMuRF warmed up.

• 190209 01:08:17 -

  • /home/keck/sch/mce_autotune.sch (still waiting on timing solution)
  • 9_kCMB_smurftest_50_015.sch - again late start due to fridge cycle

• 190209 00:42:17

  • /home/keck/sch/mce_autotune.sch (still waiting on timing solution)
  • keck36 autotune did not respond. masd restart also failed. Robert is rebooted keck36
  • Still not right. Reboot keck31. controlSystem restart

• 190208 16:40:37

  • fridgecycle.sch

• 190207 00:42:02 -

  • /home/keck/sch/mce_autotune.sch (still waiting on timing solution)
  • 9_kCMB_smurftest_50_014.sch - again late start due to fridge cycle
  • Mediator crash -- there was a series of power blips on station? -- around 1530, restarted at 20:15. However something deeper was wrong, and eventually everything on keck32 died. Restarted control system at 02:30 and all seems happy now.
  • Started in phase D but waiting for 23:00 LST - aborted schedule and started a new 9_kCMB_smurftest_50_014b.sch, with phases BCD commented out.

• 190206 16:44:03

  • fridgecycle.sch

• 190205 01:04:45 - 190206 16:44:03

  • /home/keck/sch/mce_autotune.sch (still waiting on timing solution)
  • 9_kCMB_smurftest_50_013.sch - again late start due to fridge cycle
  • runend.sch

• 190204 16:51:11

  • fridgecycle.sch

• 190203 01:35:33 - 190204 16:51:55

  • /home/keck/sch/mce_autotune.sch (still waiting on timing solution)
  • Carps in mount occasionally
  • 9_kCMB_smurftest_50_012.sch - again late start due to fridge cycle
  • Mediator crash at 0930, didn't catch until this morning at 2030. When mediator restarted, schedule resumed but was going to wait till 2300 LST to start phase D (it's only 0230), so made an abbreviated version of this schedule with phases B-D commented out. But need to wait until 0330 to start it.
  • 9_kCMB_smurftest_50_012b.sch
  • Yet another mediator crash - this time restarted control system, and then resumed schedule.

• 190202 17:12:58

  • Carps in mount
  • fridgecycle.sch

• 190130 22:50:41 - 190201 17:00:20

  • /home/keck/sch/mce_autotune.sch -- did not complete (does not wait long enough for keck34 done signal, but SMuRF logs indicate it worked anyway...working on solution)
  • Carps in mount occasionally
  • mce_autotune.sch
  • 9_kCMB_smurftest_50_011.sch
  • runend.sch

• 190130 09:47:35

  • Wasn't able to reduce 4K plate temperature by more than ~0.04 K by adjusting Rx1 compressor from 275-300 psi high side...go ahead and try another cycle (will have a few hours before starting CMB)
  • fridgecycle.sch

• 190128 01:39:00 - 190128 20:15:33

  • Fridge cycle for rx1 took longer than expected, so missed standard CMB schedule start time...
  • /home/keck/sch/mce_autotune.sch -- did not complete?
  • 9_kCMB_smurftest_50_011.sch
  • Schedule stopped in the middle of Phase C elnod because it hit soft negative az limit - didn't notice until this morning! Here's what I think happened: yesterday when I cleaned the az track during the fridge cycle, I knocked one of the az readheads out of place (so most of the az enc diffs were huge), and then during the schedule the telescope got confused and hit the limit switch.
  • Confirmed: checked enc A and the left encoder was alternating green/orange as it traveled. Replaced magnet so it's always green. Doing compressor pressure tuning today.

• 190127 17:23:21

  • fridgecycle.sch

• 190126 01:26:53 - 190127 17:20:01

  • Fridge cycle for rx1 took longer than expected, so missed standard CMB schedule start time...
  • /home/keck/sch/mce_autotune.sch -- did not complete?
  • 9_kCMB_smurftest_50_010.sch
  • runend.sch

• 190125 17:31:14

  • fridgecycle.sch

• 190124 02:26:33 - 190125 17:27:54

  • 9_kCMB_smurftest_50_009.sch
  Because of previous gcp problems, started phase B a few hours late, but otherwise attempting a full 2-day schedule
  • runend.sch

• 190123 18:24:19

  • Power cycled keck32, had to try several cycles of restarting gcp etc. to get mediator running and ssh ok.
  • runend.sch
  • fridgecycle.sch

• 190123 02:48:08

  • Carps are out of mount. Will run CMB schedule until the fridge runs out, test hold time
  • 1_kCMB_smurftest_50_008.sch
  • Mediator crash after a few hours. Tried restarting but then everything died, including ssh to keck32. Yikes.

• 190122 05:57:40 - 190122 19:04:32

  • Removed BSNS.
  • 1_kCMB_smurftest_50_007.sch, aborted in the morning because of carps in mount
  • runend.sch (forgot to run fridge cycle!)

• 190121 08:04:41 - 190121 19:57:21

  • Rotated BSNS 90 degrees, re-running mid-field beam map, then followed by CMB overnight to test system stability. Anticipate aborting in the morning because of carps in the mount.
  • config_hw_Al.sch
  • mast_raster_keck.sch
  • 1_kCMB_smurftest_50_007.sch (autoqueue, started 190121 11:35:22, aborted 190121 19:57:21)
  • runend.sch
  • fridgecycle.sch

• 190120 01:00:00

  • Found that mediator had crashed overnight. Had to restart mediator and masd. Pmac seemed to be unresponsive as well. Restarted everything and retransferred control to gcp. Now running CMB followed by runend as an autoqueue. (2_kCMB_smurftest_50_006.sch, runedn.sch). Weather is bad. Running just to check system stability.

• 190119 07:57:39

  • Starting schedule: /data/keckdaq/autoqueue/pending/1_knoise_responsivity_10_002.sch
  • Cloudy. Running to make sure nothing crashes. Will have to optimize biases for real when the weather is clear.

• 190119 06:33:48

  • schedule /home/keck/sch/etacar_raster_keck.sch
  • weather was clear at beginning of schedule but became cloudy. lower elevations see extended MAPO mast, but didn't seem much stronger than an elnod.

• 190118 07:23:10

  • schedule ~/sch/config_hw_Al.sch
  • schedule ~/sch/mast_raster_keck.sch

• 190117 20:11:18

  • Start a new mast raster, finer el steps and wider az throw
  • schedule ~/sch/mast_raster_keck.sch
  • abort_schedule, carps working in the mount!

• 190117 09:37:24

  • Installed mount extender, found source again, trying scan that should get most of the FPU
  • schedule ~/sch/mast_raster_keck.sch
  • schedule ~/sch/fridgecycle.sch

• 190117 07:42:41

  • Skies cleared up, try again
  • schedule ~/sch/etacar_raster_keck.sch

• 190117 01:45

  • Earlier started full CMB obs (only Gal phase ran). Aborted after phase finished because weather looked promising.
  • Ari found something that looks like a repeatable beam-shaped dip in Rx1. Weather is decent so trying an abbreviated schedule.
  • schedule ~/sch/etacar_raster_keck.sch

• 190114 19:53

  • Schedule ended an hour ago, playing with Rx1 He4 HS voltage to keep above 20K. Found that fridge 3,1.5 brings to 22K. Going to amend ~/gcp/antenna/hkconf/K2compact_fast_4K.cfg and run new fridge cycle, if successful will commit. Nevermind. We reverted.
  • schedule ~/sch/fridgecycle.sch

• 190113 12:00

  • cycled all fridges last night
  • Running full CMB schedule to test uMux system. 9_kCMB_smurftest_50_005.sh

• 190112 05:30:56 - 06:29:48

  • etacar_raster_keck_small.sch
  • Another Eta Carina raster, but a more fine raster over a subset of good detectors. Rx0 at bottom of drum.
  • Clear skies.
  • All rxs open to sky.

• 190110 01:22:19 - 06:33:42

  • 1_etacar_raster_keck.sch
  • Attempting to extract abs detector pointings by rastering whole FPU over Eta Carina (CarNeb in source catalog).
  • Poor weather. Overcast, high-ceiling clouds for whole schedule.
  • Rx012 open, Rx34 blanked off

• 190107 07:41:44

  • 1_knoise_responsivity_10_001.sch
  • Noise vs bias, bias points 5000 down to 200 in 100 DAC unit steps
  • Rx014 open, Rx23 blanked off
  • Begun around 4:30PM local. LMR is turned up today.
  • For some reason lasttune on band 3 failed, so this is mostly band 2.
  • Aborted 08:00:43. Realized that the mount was locked out.
  • 1_knoise_responsivity_10_002.sch
  • Aborted 08:41:00. Rx1 is slow to start acquisition, we were missing elnods.
  • Increased wait after mce_start_acq to 60s to accomodate while we track this down.

• 190106 06:08:04

  • 9_kCMB_smurftest_50_004.sch
  • Similar test schedule as before, but Rx014 open, Rx23 still blanked off
  • Last time apparently forgot to start the other 4 masd's, which is awkward.
  • Started via autoqueue successfully

• 190105 03:32

  • 9_kCMB_smurftest_50_003.sch
  • This is a test schedule with Rx1 open to the sky, the other 4 with blanking plates.
  • Runs for phase B only, then cycles the fridge. Checking to see whether we can stay tracking for the entire phase.
  • Autoqueue was unhappy for some reason, so started manually

• 181212 16:21

  • cyclefridge enable=2, on all rx

• 181123 04:17:34 -

  • 9_kCMB_smurftest_50_001.sch
  • This is a test schedule where Rx1 masd is receiving smurf fake data
  • Missed the first scanset of phase B because we started late
  • Abort schedule to proceed with Rx1 removal at 181124 00:01

• 181122 14:30

  • cyclefridge enable=2, on all rx

• 181113 21:02

  • stopped mount, locked it out. Observing done for this season

• 181112 21:17:42

  • cyclefridge enable=2, on all rx

• 181109 22:34:10

  • cyclefridge enable=2, on all rx
  • stopped CMB observations

• 181108 04:34:18 - 181109 22:34:10

  • /home/keck/sch/mce_autotune.sch
  • /data/keckdaq/autoqueue/pending/9_kCMB_50_382.sch
  • /home/keck/sch/runend.sch
  • weather: thin high clouds, blowing snow
  • windows and baffles: windows clear, quite some snow in and on the baffles

• 181107 22:42:01

  • cyclefridge enable=2, on all rx
  • He leak, B high this time, swapped for last spare
  • NO MORE SPARE LINES

• 181106 04:13:59 - 181107 22:42:01

  • /home/keck/sch/mce_autotune.sch
  • /data/keckdaq/autoqueue/pending/9_kCMB_50_381.sch
  • /home/keck/sch/runend.sch
  • weather: a bit hazym, thin high clouds
  • windows and baffles: windows clear, very little snow in the baffles

• 181105 22:49:54

  • cyclefridge enable=2, on all rx

• 181104 04:20:29 - 181105 22:49:54

  • /home/keck/sch/mce_autotune.sch
  • /data/keckdaq/autoqueue/pending/9_kCMB_50_380.sch
  • /home/keck/sch/runend.sch
  • weather: clear, thin clouds
  • windows and baffles: windows clear, very little snow in the baffles

• 181103 22:57:50

  • cyclefridge enable=2, on all rx
  • recharged comp 3
  • el current drifting apart

• 181102 04:57:53 - 181103 22:57:50

  • /home/keck/sch/mce_autotune.sch
  • /data/keckdaq/autoqueue/pending/9_kCMB_50_379.sch
  • /home/keck/sch/runend.sch
  • weather: clear, some thin clouds, clear
  • windows and baffles: clear

• 181101 23:19:38

  • cyclefridge enable=2, on all rx

• 181031 04:11:28 - 181101 23:19:38

  • /home/keck/sch/mce_autotune.sch
  • /data/keckdaq/autoqueue/pending/9_kCMB_50_378.sch
  • 181031 08:21:26 rx1 is cold and MCE1 and masd1 turned back on
  • mediator stopped
  • restarted gcp, masd
  • 181031 08:25:10 Starting schedule: /home/keck/sch/mce_autotune.sch
  • 181031 08:26:59 Starting schedule: /data/keckdaq/autoqueue/pending/9_kCMB_50_378.sch
  • 181101 14:35 masd0 stopped
  • restarted masd0, didn't do anything, rebooted keck33
  • rx1 warmed up a bit early
  • /home/keck/sch/runend1.sch
  • weather: mostly clear
  • windows and baffles: clear

• 181030 23:13:30

  • cyclefridge enable=2, on all rx
  • recharged comp 2, valve is very dodgy, only needs about 3 degrees to fully charge
  • after charging it continued to leak He into comp and shut off due to high He
  • stopped cycle on rx1, and let it cool down
  • started new cycle on rx1, will be a few scansets late

• 181029 06:30:21 - 181030 23:13:30

  • run started late due to problems with fridge cycle, see below
  • /home/keck/sch/mce_autotune.sch
  • /data/keckdaq/autoqueue/pending/9_kCMB_50_377.sch
  • massive He leak in D high (rx3) swapped to spare line
  • /home/keck/sch/runend.sch
  • weather: thin clouds, mostly clear
  • windows and baffles: clear

• 181028 23:21:27

  • cyclefridge enable=2, on all rx
  • 181028 23:57:37 Starting schedule: /home/keck/sch/starpoint_pole_keck_summer.sch
  • 181029 01:25:22 Exiting schedule: /home/keck/sch/starpoint_pole_keck_summer.sch
  • was quite tough, about half of the stars I could only see on the monitor not the frame grabber
  • abnormal fridge cycle on all rx, except rx1, maybe due to starpointing
  • aborted cycle on all, except rx1

• 181027 05:21:29 - 181028 23:21:27

  • /home/keck/sch/mce_autotune.sch
  • /data/keckdaq/autoqueue/pending/9_kCMB_50_376.sch
  • 181028 04:26 mediator stopped
  • restarted gcp, masd
  • pmac was throwing an exeception, restarted keck31
  • restarted gcp, masd
  • 181028 05:30:29 Starting schedule: /home/keck/sch/mce_autotune.sch
  • 181028 05:32:18 Starting schedule: /data/keckdaq/autoqueue/pending/9_kCMB_50_376.sch
  • /home/keck/sch/runend.sch
  • weather: partly cloudy, high clouds, overcast, clearing
  • windows and baffles: clear

• 181026 23:29:15

  • cyclefridge enable=2, on all rx

• 181025 05:29:21 - 181026 23:29:15

  • /home/keck/sch/mce_autotune.sch
  • /data/keckdaq/autoqueue/pending/9_kCMB_50_375.sch
  • /home/keck/sch/runend.sch
  • weather: thin clouds, blowing snow, partly cloudy
  • windows and baffles: windows clear, very little snow on the baffles

• 181024 23:37:05

  • cyclefridge enable=2, on all rx

• 181023 05:37:12 - 181024 23:37:05

  • /home/keck/sch/mce_autotune.sch
  • /data/keckdaq/autoqueue/pending/9_kCMB_50_374.sch
  • /home/keck/sch/runend.sch
  • weather: hazy, thin clouds
  • windows and baffles: windows clear, very little snow on the baffles

• 181022 23:44:56

  • cyclefridge enable=2, on all rx
  • replaced bad line with an old spare
  • el current drifting apart

• 181021 05:45:04 - 181022 23:44:56

  • /home/keck/sch/mce_autotune.sch
  • /data/keckdaq/autoqueue/pending/9_kCMB_50_373.sch
  • /home/keck/sch/runend.sch
  • weather: thin clouds, hazy
  • windows and baffles: clear

• 181020 23:52:47

  • cyclefridge enable=2, on all rx

• 181019 05:52:56 - 181020 23:52:47

  • /home/keck/sch/mce_autotune.sch
  • /data/keckdaq/autoqueue/pending/9_kCMB_50_372.sch
  • /home/keck/sch/runend.sch
  • weather: thin clouds, overcast
  • windows and baffles: clear

• 181019 00:00:42

  • cyclefridge enable=2, on all rx
  • recharged comp 2

• 181017 05:59:40 - 181019 00:00:42

  • /home/keck/sch/mce_autotune.sch
  • /data/keckdaq/autoqueue/pending/9_kCMB_50_371.sch
  • 181018 08:45 massive leak in A high (rx0) swapped for installed spare line
  • brief warm up of rx0 just over 300mK
  • /home/keck/sch/runend.sch
  • weather: overcast, thin high clouds
  • windows and baffles: windows clear, lots of snow in and on the baffles

• 181017 00:08:38

  • cyclefridge enable=2, on all rx
  • recharged comp 4
  • replaced faulty line with new spare

• 181015 06:03:14 - 181017 00:08:38

  • /home/keck/sch/mce_autotune.sch
  • /data/keckdaq/autoqueue/pending/9_kCMB_50_370.sch
  • /home/keck/sch/runend.sch
  • weather: thin clouds, clear, overcast
  • windows and baffles: windows clear, very little snow in and on the baffles

• 181015 00:16:23

  • cyclefridge enable=2, on all rx
  • recharged comp 1, 3

• 181013 06:16:32 - 181015 00:16:23

  • /home/keck/sch/mce_autotune.sch
  • /data/keckdaq/autoqueue/pending/9_kCMB_50_369.sch
  • massive He leak in C high, slight warming of FPU on rx2
  • swapped to spare line
  • /home/keck/sch/runend.sch
  • weather: overcast, clear, thin clouds
  • windows and baffles: windows clear, very little snow in the baffles

• 181013 00:37:16

  • cyclefridge enable=2, on all rx

• 181011 06:07:44 - 181013 00:24:16

  • /home/keck/sch/mce_autotune.sch
  • /data/keckdaq/autoqueue/pending/9_kCMB_50_368.sch
  • 181012 00:45:38 suspend_schedule
  • az-diff jump, replaced A4 readhead roller
  • 181012 01:53:54 resume_schedule
  • /home/keck/sch/runend.sch
  • weather: overcast, blowing snow, warm
  • windows and baffles: windows clear, a lot of snow in the baffles

• 181011 00:32:08

  • cyclefridge enable=2, on all rx

• 181009 06:09:33 - 181011 00:32:08

  • /home/keck/sch/mce_autotune.sch
  • /data/keckdaq/autoqueue/pending/9_kCMB_50_367.sch
  • /home/keck/sch/runend.sch
  • weather: overcast, blowing snow, warm
  • windows and baffles: windows clear, tons of snow in the baffles

• 181009 00:40:00

  • cyclefridge enable=2, on all rx
  • replaced bad he line with new spare, used all 12 new lines
  • adjusted readhead A1

• 181007 06:23:13 - 181009 00:40:00

  • /home/keck/sch/mce_autotune.sch
  • /data/keckdaq/autoqueue/pending/9_kCMB_50_366.sch
  • small az-diff jump, rehomed, data are good until jump and after resume
  • 181008 14:05:54 resume_schedule
  • /home/keck/sch/runend.sch
  • weather: hazy, overcast, blowing snow, warm
  • windows and baffles: windows clear, some snow in the baffles

• 181007 00:47:50

  • cyclefridge enable=2, on all rx
  • He leak in B low, replaced with spare line
  • this was one of the original 2014 lines

• 181005 06:28:49 - 181007 00:47:50

  • /home/keck/sch/mce_autotune.sch
  • /data/keckdaq/autoqueue/pending/9_kCMB_50_365.sch
  • /home/keck/sch/runend.sch
  • weather: overcast, hazy
  • windows and baffles: windows clear, some snow in the baffles

• 181005 00:55:44

  • cyclefridge enable=2, on all rx

• 181003 06:24:41 - 181005 00:55:44

  • /home/keck/sch/mce_autotune.sch
  • /data/keckdaq/autoqueue/pending/9_kCMB_50_364.sch
  • /home/keck/sch/runend.sch
  • weather: overcast, clearing, clear, overcast
  • windows and baffles: windows clear, little bit of snow in and on the baffles

• 181003 01:03:34

  • cyclefridge enable=2, on all rx
  • recharged comp 5

• 181001 07:07:51 - 181003 01:03:34

  • /home/keck/sch/mce_autotune.sch
  • /data/keckdaq/autoqueue/pending/9_kCMB_50_363.sch
  • /home/keck/sch/runend.sch
  • weather: overcast, clear, overcast
  • windows and baffles: windows clear, some snow in and on the baffles

• 181001 01:11:27

  • cyclefridge enable=2, on all rx

• 180929 06:38:32 - 181001 01:11:27

  • /home/keck/sch/mce_autotune.sch
  • /data/keckdaq/autoqueue/pending/9_kCMB_50_362.sch
  • /home/keck/sch/runend.sch
  • weather: overcast
  • windows and baffles: windows clear, some snow in and on the baffles

• 180929 01:19:20

  • cyclefridge enable=2, on all rx

• 180927 06:45:13 - 180929 01:19:20

  • /home/keck/sch/mce_autotune.sch
  • /data/keckdaq/autoqueue/pending/9_kCMB_50_361.sch
  • /home/keck/sch/runend.sch
  • weather: overcast,clear, overcast
  • windows and baffles: windows with a 3 inch center patch of snow, lots of snow in and on the baffles

• 180927 01:27:10

  • cyclefridge enable=2, on all rx
  • recharged comp 4

• 180925 06:58:28 - 180927 01:27:10

  • /home/keck/sch/mce_autotune.sch
  • /data/keckdaq/autoqueue/pending/9_kCMB_50_360.sch
  • mediator stopped
  • restarted gcp/masd
  • 180925 11:10:28 Starting schedule: /home/keck/sch/mce_autotune.sch
  • 180925 11:12:18 Starting schedule: /data/keckdaq/autoqueue/pending/9_kCMB_50_360.sch
  • /home/keck/sch/runend.sch
  • weather: thin clouds, overcast
  • windows and baffles: clear

• 180925 01:35:03

  • cyclefridge enable=2, on all rx
  • recharged comp 3

• 180923 07:19:48 - 180925 01:35:03

  • /home/keck/sch/mce_autotune.sch
  • /data/keckdaq/autoqueue/pending/9_kCMB_50_359.sch
  • mediator stopped
  • restarted gcp/masd
  • 180923 10:36:00 Starting schedule: /home/keck/sch/mce_autotune.sch
  • 180923 10:37:50 Starting schedule: /data/keckdaq/autoqueue/pending/9_kCMB_50_359.sch
  • /home/keck/sch/runend.sch
  • weather: mostly clear overhead, hazy, blowing snow, thin clouds
  • windows and baffles: windows clear, snow in and on the baffles

• 180923 01:42:54

  • cyclefridge enable=2, on all rx
  • recharged comp 2

• 180921 07:08:38 - 180923 01:42:54

  • /data/keckdaq/autoqueue/pending/9_kCMB_50_358.sch
  • /home/keck/sch/runend.sch
  • weather: overcast, blowing snow, clearing
  • windows and baffles: windows clear, some snow in and on the baffles

• 180921 01:50:46

  • cyclefridge enable=2, on all rx

• 180919 07:50:53 - 180921 01:50:46

  • /home/keck/sch/mce_autotune.sch
  • /data/keckdaq/autoqueue/pending/9_kCMB_50_357.sch
  • /home/keck/sch/runend.sch
  • weather: clear, blowing snow, hazy
  • windows and baffles: windows clear, some snow in and on the baffles

• 180919 01:58:37

  • cyclefridge enable=2, on all rx

• 180917 07:25:38 - 180919 01:58:37

  • /home/keck/sch/mce_autotune.sch
  • /data/keckdaq/autoqueue/pending/9_kCMB_50_356.sch
  • 180918 22:20 fan belt in airhandler broke, all compressors shut off
  • replaced fan belt
  • all rx cooling down again
  • 180918 23:32:04 Starting schedule: /home/keck/sch/mce_autotune.sch
  • 180918 23:33:53 Starting schedule: /data/keckdaq/autoqueue/pending/9_kCMB_50_356.sch
  • /home/keck/sch/runend.sch
  • weather: clear
  • windows and baffles: clear

• 180917 02:06:28

  • cyclefridge enable=2, on all rx

• 180915 07:48:56 - 180917 02:06:28

  • /home/keck/sch/mce_autotune.sch
  • /data/keckdaq/autoqueue/pending/9_kCMB_50_355.sch
  • /home/keck/sch/runend.sch
  • weather: clear above
  • windows and baffles: clear

• 180915 02:14:22

  • cyclefridge enable=2, on all rx
  • fridge cycle nominal again, rx0 is behaving normal again

• 180913 14:28:57 - 180915 02:14:22

  • /home/keck/sch/mce_autotune.sch
  • /data/keckdaq/autoqueue/pending/9_kCMB_50_354.sch
  • rx0 not totally cooling down, although UC is at 260mK
  • after several hours FPU rx0 is sitll at 322mK, UC at 260mK
  • powering mce0 off, rx0 temperature drops slowly
  • 180914 00:20:29 rx0 cooled down to below 300mK
  • started masd0
  • /home/keck/sch/runend.sch
  • weather: mostly clear, clear
  • windows and baffles: clear

• 180913 09:36:25

  • cyclefridge enable=2, on all rx
  • 180913 10:22:16 Starting schedule: /home/keck/sch/starpoint_pole_keck_summer.sch
  • 180913 12:02:34 Exiting schedule: /home/keck/sch/starpoint_pole_keck_summer.sch
  • pmac threw exeption when using slew command !?
  • restarted gcp masd
  • keck33 unresponsive, restarted all kecks
  • tracker window didn't update pmac data
  • rebooted keck31

• 180913 08:14:29

  • accidentely started another fridgecycle, aborted schedules, stopped cycle
  • 4Ks warmed up quickly
  • restarted GCP and masd
  • pmac through exception, after another GCP restart, pmac is behaving fine again
  • blew IC and UC, therefore starting another fridgecycle

• 180913 02:22:13

  • cyclefridge enable=2, on all rx
  • bit of a unusual fridgecycle, rx=0 was not the first to finish
  • and rx2 was the slowest, although He pressures are good!?

• 180911 10:24:53 - 180913 02:22:13

  • /home/keck/sch/mce_autotune.sch
  • /data/keckdaq/autoqueue/pending/9_kCMB_50_353.sch
  • /home/keck/sch/runend.sch
  • weather:mostly clear, thin clouds, clear
  • windows and baffles: clear

• 180911 05:07:40

  • cyclefridge enable=2, on all rx
  • I did not start runend and fridgecycle in time, my fault, sorry !!
  • therefore run 353 will start late

• 180909 07:58:22 - 180911 05:07:40

  • /home/keck/sch/mce_autotune.sch
  • /data/keckdaq/autoqueue/pending/9_kCMB_50_352.sch
  • /home/keck/sch/runend.sch
  • weather: clear
  • windows and baffles: clear

• 180909 02:37:55

  • cyclefridge enable=2, on all rx
  • recharged comp 4

• 180907 08:28:53 - 180909 02:37:55

  • /home/keck/sch/mce_autotune.sch
  • /data/keckdaq/autoqueue/pending/9_kCMB_50_351.sch
  • /home/keck/sch/runend.sch
  • weather: clear
  • windows and baffles: clear

• 180907 02:45:49

  • cyclefridge enable=2, on all rx
  • recharged comp 2

• 180905 08:45:56 - 180907 02:45:49

  • /home/keck/sch/mce_autotune.sch
  • /data/keckdaq/autoqueue/pending/9_kCMB_50_350.sch
  • /home/keck/sch/runend.sch
  • weather: clear, cold, mostly clear
  • windows and baffles: windows clear, some snow on the baffles

• 180905 02:53:41

  • cyclefridge enable=2, on all rx

• 180903 09:01:57 - 180905 02:53:41

  • /home/keck/sch/mce_autotune.sch
  • /data/keckdaq/autoqueue/pending/9_kCMB_50_349.sch
  • mediator stopped
  • restarted gcp/masd
  • 180904 08:49:25 Starting schedule: /home/keck/sch/mce_autotune.sch
  • 180904 08:51:15 Starting schedule: /data/keckdaq/autoqueue/pending/9_kCMB_50_349.sch
  • /home/keck/sch/runend.sch
  • weather: clear, cold
  • windows and baffles: clear

• 180903 03:01:34

  • cyclefridge enable=2, on all rx

• 180901 09:01:40 - 180903 03:01:34

  • /home/keck/sch/mce_autotune.sch
  • /data/keckdaq/autoqueue/pending/9_kCMB_50_348.sch
  • /home/keck/sch/runend.sch
  • weather: overcast, blowing snow, clearing, clear
  • windows and baffles: windows clear, quite some snow in the baffles

• 180901 03:09:23

  • cyclefridge enable=2, on all rx

• 180830 08:54:25 - 180901 03:09:23

  • /home/keck/sch/mce_autotune.sch
  • /data/keckdaq/autoqueue/pending/9_kCMB_50_347.sch
  • /home/keck/sch/runend.sch
  • weather: hazy, mostly clear, clear, getting windy, blowing snow, overcast
  • windows and baffles: windows clear, ton of snow in and on the baffles

• 180830 03:17:19

  • cyclefridge enable=2, on all rx
  • recharged comp 2, 3

• 180828 09:17:23 - 180830 03:17:19

  • /home/keck/sch/mce_autotune.sch
  • /data/keckdaq/autoqueue/pending/9_kCMB_50_346.sch
  • 180828 15:28 small az-diff jump over 30k, suspend schedule, could not see anything wrong
  • 180828 15:41:57 resume_schedule
  • 180828 16:47 az-diff jump, readjusted readhead 2A
  • 180828 17:04:42 resume_schedule
  • /home/keck/sch/runend.sch
  • weather: clear, mostly clear, hazy
  • windows and baffles: windows clear, quite some snow in the baffles

• 180828 03:25:16

  • cyclefridge enable=2, on all rx

• 180826 09:15:20 - 180828 03:25:16

  • /home/keck/sch/mce_autotune.sch
  • /data/keckdaq/autoqueue/pending/9_kCMB_50_345.sch
  • /home/keck/sch/runend.sch
  • weather: mostly clear
  • windows and baffles: clear

• 180826 03:33:00

  • cyclefridge enable=2, on all rx

• 180824 09:32:42 - 180826 03:33:00

  • /home/keck/sch/mce_autotune.sch
  • /data/keckdaq/autoqueue/pending/9_kCMB_50_344.sch
  • /home/keck/sch/runend.sch
  • weather: mostly clear
  • windows and baffles: clear

• 180824 03:40:53

  • cyclefridge enable=2, on all rx
  • 180824 05:34:21 Starting schedule: /home/keck/sch/starpoint_pole_keck_winter.sch
  • 180824 09:30:53 Exiting schedule: /home/keck/sch/starpoint_pole_keck_winter.sch
  • swapped bad He-line for new spare

• 180822 09:19:48 - 180824 03:40:53

  • /home/keck/sch/mce_autotune.sch
  • keck 33 froze restarted keck33 and masd
  • /data/keckdaq/autoqueue/pending/9_kCMB_50_343.sch
  • /home/keck/sch/runend.sch
  • weather: hazy,, mostly clear
  • windows and baffles: windows clear, very little snow on the baffles

• 180822 03:52:02

  • cyclefridge enable=2, on all rx
  • recharged comp 4

• 180820 09:41:52 - 180822 03:52:02

  • /home/keck/sch/mce_autotune.sch
  • /data/keckdaq/autoqueue/pending/9_kCMB_50_342.sch
  • /home/keck/sch/runend.sch
  • weather: very hazy, thin clouds, hazy
  • windows and baffles: clear

• 180820 03:56:37

  • cyclefridge enable=2, on all rx
  • 180820 04:44:43 Starting schedule: /home/keck/sch/starpoint_pole_keck_winter.sch
  • 180820 04:50:44 abort_schedule, too hazy

• 180818 09:56:42 - 180820 03:56:37

  • /home/keck/sch/mce_autotune.sch
  • /data/keckdaq/autoqueue/pending/9_kCMB_50_341.sch
  • /home/keck/sch/runend.sch
  • weather: mostly clear, little hazy
  • windows and baffles: clear

• 180818 04:04:27

  • cyclefridge enable=2, on all rx
  • leak in E low, replaced with spare line

• 180816 10:04:34 - 180818 04:04:27

  • /home/keck/sch/mce_autotune.sch
  • /data/keckdaq/autoqueue/pending/9_kCMB_50_340.sch
  • mediator stopped
  • restarted gcp/masd
  • 180817 07:52:45 Starting schedule: /home/keck/sch/mce_autotune.sch
  • 180817 07:54:35 Starting schedule: /data/keckdaq/autoqueue/pending/9_kCMB_50_340.sch
  • /home/keck/sch/runend.sch
  • weather: mostly clear
  • windows and baffles: clear

• 180816 04:11:56

  • cyclefridge enable=2, on all rx
  • recharged comp 5, possible leak

• 180814 09:41:14 - 180816 04:11:56

  • /home/keck/sch/mce_autotune.sch
  • /data/keckdaq/autoqueue/pending/9_kCMB_50_339.sch
  • 180815 18:40 masd0 stopped, restarted keck33 and masd0
  • /home/keck/sch/runend.sch
  • weather: hazy, thin clouds, mostly clear
  • windows and baffles: clear

• 180814 04:20:13

  • cyclefridge enable=2, on all rx
  • replaced bad line with new spare line

• 180812 10:18:13 - 180814 04:20:13

  • /home/keck/sch/mce_autotune.sch
  • /data/keckdaq/autoqueue/pending/9_kCMB_50_338.sch
  • /home/keck/sch/runend.sch
  • weather: hazy, blowing snow, high winds
  • windows and baffles: windows clear, some snow on the baffles

• 180812 04:28:03

  • cyclefridge enable=2, on all rx
  • recharged comp 2
  • leak in A high, replaced with spare
  • that is the first of the new lines going bad, was in for 5 months

• 180810 10:07:52 - 180812 04:28:03

  • /home/keck/sch/mce_autotune.sch
  • /data/keckdaq/autoqueue/pending/9_kCMB_50_337.sch
  • /home/keck/sch/runend.sch
  • weather: hazy, blowing snow, high winds
  • windows and baffles: windows clear, very little snow on the baffles

• 180810 04:35:54

  • cyclefridge enable=2, on all rx
  • readjusted readhead 3A

• 180808 10:23:30 - 180810 04:35:54

  • /home/keck/sch/mce_autotune.sch
  • /data/keckdaq/autoqueue/pending/9_kCMB_50_336.sch
  • az diff jumps, some problems with readhead 3A
  • reject data once az gets over 30k until resume schedule as follows
  • 180809 10:17:26 resume_schedule
  • 180809 12:27:20 resume_schedule
  • 180809 20:10:51 resume_schedule
  • /home/keck/sch/runend.sch
  • weather: mostly clear, quite hazy
  • windows and baffles: clear

• 180808 04:43:45

  • cyclefridge enable=2, on all rx

• 180806 10:15:30 - 180808 04:43:45

  • /home/keck/sch/mce_autotune.sch
  • /data/keckdaq/autoqueue/pending/9_kCMB_50_335.sch
  • /home/keck/sch/runend.sch
  • weather: cloudy, blowing snow, clearing
  • windows and baffles: clear

• 180806 04:51:40

  • cyclefridge enable=2, on all rx

• 180804 10:51:45 - 180806 04:51:40

  • /home/keck/sch/mce_autotune.sch
  • /data/keckdaq/autoqueue/pending/9_kCMB_50_334.sch
  • /home/keck/sch/runend.sch
  • weather: hazy, thin clouds, blowing snow
  • windows and baffles: windows clear, some snow on the baffles

• 180804 04:59:30

  • cyclefridge enable=2, on all rx

• 180802 10:34:36 - 180804 04:59:30

  • /home/keck/sch/mce_autotune.sch
  • /data/keckdaq/autoqueue/pending/9_kCMB_50_333.sch
  • /home/keck/sch/runend.sch
  • weather: little hazy, blowing snow, mostly clear
  • windows and baffles: clear

• 180802 05:07:21

  • cyclefridge enable=2, on all rx
  • recharged comp 4

• 180731 11:00:57 - 180802 05:07:21

  • /home/keck/sch/mce_autotune.sch
  • /data/keckdaq/autoqueue/pending/9_kCMB_50_332.sch
  • /home/keck/sch/runend.sch
  • weather: mostly clear, hazy, blowing snow
  • windows and baffles: windows clear, very little snow on the baffles

• 180731 05:31:49

  • cyclefridge enable=2, on all rx
  • recharged comp 3, 5

• 180729 11:25:32 - 180731 05:31:49

  • /home/keck/sch/mce_autotune.sch
  • /data/keckdaq/autoqueue/pending/9_kCMB_50_331.sch
  • el current drifting apart
  • /home/keck/sch/runend.sch
  • weather: mostly clear, hazy, clearing
  • windows and baffles: clear

• 180729 05:23:06

  • cyclefridge enable=2, on all rx

• 180727 11:23:12 - 180729 05:23:06

  • /home/keck/sch/mce_autotune.sch
  • /data/keckdaq/autoqueue/pending/9_kCMB_50_330.sch
  • mediator stopped
  • restarted gcp/masd
  • 180728 15:09:55 Starting schedule: /home/keck/sch/mce_autotune.sch
  • 180728 15:11:45 Starting schedule: /data/keckdaq/autoqueue/pending/9_kCMB_50_330.sch
  • /home/keck/sch/runend.sch
  • weather: hazy, blowing snow, clearing
  • windows and baffles: windows clear, barely any snow in the baffles

• 180727 05:30:59

  • cyclefridge enable=2, on all rx
  • recharged comp 2

• 180725 11:01:13 - 180727 05:30:59

  • /home/keck/sch/mce_autotune.sch
  • /data/keckdaq/autoqueue/pending/9_kCMB_50_329.sch
  • /home/keck/sch/runend.sch
  • weather: hazy, blowing snow, mostly clear
  • windows and baffles: windows clear, very little snow on the baffles

• 180725 05:38:47

  • cyclefridge enable=2, on all rx

• 180723 12:47:17 - 180725 05:38:47

  • /home/keck/sch/mce_autotune.sch
  • /data/keckdaq/autoqueue/pending/9_kCMB_50_328.sch
  • problems with masd1, keck34, turns out disk space on keck34/data was full
  • 180723 16:06:59 abort_schedule
  • 180723 16:07:23 Starting schedule: /home/keck/sch/mce_autotune.sch
  • 180723 16:09:12 Starting schedule: /data/keckdaq/autoqueue/pending/9_kCMB_50_328.sch
  • all kecks are happy
  • /home/keck/sch/runend.sch
  • weather: hazy, blowing snow
  • windows and baffles: windows clear, quite some snow in the baffles

• 180723 05:48:56

  • cyclefridge enable=2, on all rx
  • problems starting new cmb, schedule will quite because of MCE reconfiguration failed!
  • restarted sync box and all keck

• 180721 14:32:57 - 180723 05:46:21

  • later runstart since I was trying to fixc pmac issue
  • /home/keck/sch/mce_autotune.sch
  • /data/keckdaq/autoqueue/pending/9_kCMB_50_327.sch
  • many more pmac failures
  • mediator stopped
  • restarted gcp/masd
  • 180721 23:17:41 Starting schedule: /home/keck/sch/mce_autotune.sch
  • 180721 23:19:31 Starting schedule: /data/keckdaq/autoqueue/pending/9_kCMB_50_327.sch
  • el current drifting apart
  • open the loops with "k" didn't work
  • masd1 stopped
  • restarted keck34
  • masd1 was a bit slow, after every scan start and would trip the pager
  • disabled pager
  • looks like rx1 wasn't recording from examiner plots
  • /home/keck/sch/runend.sch
  • weather: clear
  • windows and baffles: clear

• 180721 04:58:42

  • pmac giving ERR012
  • cold reboot of all keck computers
  • cyclefridge enable=2, on all rx
  • 180721 06:07:51 Starting schedule: /home/keck/sch/starpoint_pole_keck_winter.sch
  • 180721 10:01:58 Exiting schedule: /home/keck/sch/starpoint_pole_keck_winter.sch
  • pmac is behaving strangly, checked all connections in keck31 and hand controller, but still
  • problems
  • installed pmac spare, but that one doesn't work
  • swapped card several times but always same result, spare doesn't work, current card gives problems

• 180719 11:28:15 - 180721 04:35:08

  • /home/keck/sch/mce_autotune.sch
  • /data/keckdaq/autoqueue/pending/9_kCMB_50_326.sch
  • 180720 01:07:36 pmac stopped !??
  • mulitple pmac stops
  • rebooted keck31
  • 180720 09:28 three power outages
  • mediator stopped
  • restarted gcp/masd
  • 180720 14:17:57 Starting schedule: /home/keck/sch/mce_autotune.sch
  • 180720 14:20:02 Starting schedule: /data/keckdaq/autoqueue/pending/9_kCMB_50_326.sch
  • /home/keck/sch/runend.sch
  • pmac stopped again
  • aborted run about 1h before acutal runend to work on pmac
  • weather: clear, very hazy, mostly clear, clear
  • windows and baffles: clear

• 180719 06:02:24

  • cyclefridge enable=2, on all rx

• 180717 12:02:32 - 180719 06:02:24

  • /home/keck/sch/mce_autotune.sch
  • /data/keckdaq/autoqueue/pending/9_kCMB_50_325.sch
  • /home/keck/sch/runend.sch
  • weather: mostly clear, blowing snow
  • windows and baffles: clear

• 180717 06:10:16

  • cyclefridge enable=2, on all rx
  • installed new az He line

• 180715 12:10:23 - 180717 06:10:16

  • /home/keck/sch/mce_autotune.sch
  • /data/keckdaq/autoqueue/pending/9_kCMB_50_324.sch
  • 180716 11:48:43 suspend_schedule
  • az-diff jump, just above 30k, readjusted readhead 1A
  • 180716 12:00:06 resume_schedule
  • /home/keck/sch/runend.sch
  • weather: clear, mostly clear, blowing snow
  • windows and baffles: windows clear, very little snow on the baffles

• 180715 06:18:08

  • cyclefridge enable=2, on all rx
  • recharged comp 2, 3, 4

• 180713 12:18:15 - 180715 06:18:08

  • /home/keck/sch/mce_autotune.sch
  • /data/keckdaq/autoqueue/pending/9_kCMB_50_323.sch
  • /home/keck/sch/runend.sch
  • problems with signalhound1
  • weather: very hazy<, clearing, clear/li>
  • windows and baffles: windows clear, very little snow in the baffles

• 180713 06:26:04

  • cyclefridge enable=2, on all rx
  • leak in D high
  • replaced faulty line with spare

• 180711 11:56:26 - 180713 06:26:04

  • /home/keck/sch/mce_autotune.sch
  • /data/keckdaq/autoqueue/pending/9_kCMB_50_322.sch
  • /home/keck/sch/runend.sch
  • weather: clear, blowing snow, very hazy
  • windows and baffles: windows clear, quite some snow in and on the baffles

• 180711 06:33:50

  • cyclefridge enable=2, on all rx
  • replaced bad He line with new spare

• 180709 12:38:45 - 180711 06:33:50

  • /home/keck/sch/mce_autotune.sch
  • /data/keckdaq/autoqueue/pending/9_kCMB_50_321.sch
  • /home/keck/sch/runend.sch
  • weather: mostly clear, clear
  • windows and baffles: clear

• 180709 06:41:46

  • cyclefridge enable=2, on all rx
  • recharged rx2, high side was still at 287psi, but possible leak,
  • slight warm up 13h before runend
  • readheads look good, freak az-diff jump coudn't be reproduced
  • replaced C low with spare line

• 180707 12:09:26 - 180709 06:41:46

  • /home/keck/sch/mce_autotune.sch
  • /data/keckdaq/autoqueue/pending/9_kCMB_50_320.sch
  • huge az-diff jump on the last scans of last scanset
  • let run finish
  • furnace quite in MAPO, got just below freezing downstairs
  • /home/keck/sch/runend.sch
  • weather: clear, cold
  • windows and baffles: clear

• 180707 06:49:36

  • cyclefridge enable=2, on all rx

• 180705 12:06:18 - 180707 06:49:36

  • /home/keck/sch/mce_autotune.sch
  • /data/keckdaq/autoqueue/pending/9_kCMB_50_319.sch
  • /home/keck/sch/runend.sch
  • weather: hazy, blowing snow, clearing, getting cold, very clear
  • windows and baffles: windows clear, quite a bit of snow on and in the baffles

• 180705 06:31:02

  • cyclefridge enable=2, on all rx
  • recharged comp 2, 5

• 180703 12:32:22 - 180705 06:31:02

  • /home/keck/sch/mce_autotune.sch
  • /data/keckdaq/autoqueue/pending/9_kCMB_50_318.sch
  • 180704 04:39:52 suspend_schedule
  • az-diff jump, readjusted readhead A2
  • 180704 05:09:07 resume_schedule
  • schedule didn't pick up at the next scan set !?
  • 180704 05:27:14 abort_schedule
  • 180704 05:27:23 Starting schedule: /data/keckdaq/autoqueue/pending/9_kCMB_50_318.sch
  • mediator stopped
  • restarted gcp/masd
  • since there were only a few minutes left until runend, started fridgcycle right away
  • weather: very hazy, clearing, clear, hazy, wind picking up
  • windows and baffles: windows clear, ton of snow in and on the baffles

• 180703 07:05:18

  • cyclefridge enable=2, on all rx

• 180701 13:05:26 - 180703 07:05:18

  • /home/keck/sch/mce_autotune.sch
  • /data/keckdaq/autoqueue/pending/9_kCMB_50_317.sch
  • /home/keck/sch/runend.sch
  • weather: clear, very hazy, wind picking up
  • windows and baffles: windows 3in snow patch in center, lots of snow in the baffles

• 180701 07:31:00

  • cyclefridge enable=2, on all rx

• 180629 13:46:11 - 180701 07:30:18

  • /home/keck/sch/mce_autotune.sch
  • /data/keckdaq/autoqueue/pending/9_kCMB_50_316.sch
  • mediator stopped
  • restarted gcp/masd
  • 180630 05:53:42 Starting schedule: /home/keck/sch/mce_autotune.sch
  • 180630 05:56:33 Starting schedule: /data/keckdaq/autoqueue/pending/9_kCMB_50_316.sch
  • mediator stopped
  • restarted gcp/masd
  • 180701 07:30:23 abort_schedule
  • weather: clear
  • windows and baffles: clear

• 180629 07:43:08

  • cyclefridge enable=2, on all rx
  • recharged comp 3
  • but still no leak could be found

• 180627 12:38:16 - 180629 07:30:07

  • /home/keck/sch/mce_autotune.sch
  • /data/keckdaq/autoqueue/pending/9_kCMB_50_315.sch
  • 180628 23:05 masd2 stopped, pager initiated
  • came back after reboot of keck35
  • /home/keck/sch/runend.sch
  • weather: haze, clear
  • windows and baffles: clear

• 180627 07:19:24

  • cyclefridge enable=2, on all rx

• 180625 13:29:02 - 180627 07:17:10

  • /home/keck/sch/mce_autotune.sch
  • /data/keckdaq/autoqueue/pending/9_kCMB_50_314.sch
  • mediator stopped
  • restarted gcp/masd
  • 180627 04:44:38 Starting schedule: /home/keck/sch/mce_autotune.sch
  • 180627 04:46:28 Starting schedule: /data/keckdaq/autoqueue/pending/9_kCMB_50_314.sch
  • mediator stopped
  • restarted gcp/masd
  • aborted run, since there were only a few minutes to go

  • ARA (Icecube) did some testing:
    
    Hello Dark Sector!
    Yesterday we performed a ARA deep pulsar run during which we switched on some white noise generators 1450m below he ice.
    
    Here are the exact times this happened:
    
    Started ARA pulsar at 2018-06-25 21:28:46.365974 UTC
    Stopped ARA Pulsar at 2018-06-25 21:43:46.826194 UTC
    Started ARA pulsar at 2018-06-25 22:02:57.754798 UTC
    Stopped ARA Pulsar at 2018-06-25 22:17:58.166578 UTC
    Started ARA pulsar at 2018-06-25 22:23:42.940100 UTC
    Stopped ARA Pulsar at 2018-06-25 22:53:43.361662 UTC
    

  • weather: clear, haze
  • windows and baffles: clear

• 180625 07:36:46

  • cyclefridge enable=2, on all rx
  • recharged comp 2,3

• 180623 13:20:43 - 180625 07:36:46

  • /home/keck/sch/mce_autotune.sch
  • /data/keckdaq/autoqueue/pending/9_kCMB_50_313.sch
  • mediator stopped
  • restarted gcp/masd
  • 180625 04:08:35 Starting schedule: /home/keck/sch/mce_autotune.sch
  • 180625 04:10:26 Starting schedule: /data/keckdaq/autoqueue/pending/9_kCMB_50_313.sch
  • /home/keck/sch/runend.sch
  • weather: clear
  • windows and baffles: clear

• 180623 07:44:37

  • cyclefridge enable=2, on all rx
  • 180623 09:46:46 Starting schedule: /home/keck/sch/starpoint_pole_keck_winter.sch
  • 180623 13:20:43 Exiting schedule: /home/keck/sch/starpoint_pole_keck_winter.sch

• 180621 13:26:05 - 180623 07:44:37

  • /home/keck/sch/mce_autotune.sch
  • /data/keckdaq/autoqueue/pending/9_kCMB_50_312.sch
  • /home/keck/sch/runend.sch
  • weather: clear
  • windows and baffles: clear

• 180621 08:02:11

  • cyclefridge enable=2, on all rx
  • replaced bad line with new spare, no pressure loss in comp 3 over the past 2 days ??

• 180619 13:52:37 - 180621 08:02:11

  • /home/keck/sch/mce_autotune.sch
  • /data/keckdaq/autoqueue/pending/9_kCMB_50_311.sch
  • /home/keck/sch/runend.sch
  • weather: clearing clear
  • windows and baffles: windows clear, very little snow in the baffles

• 180619 08:00:20

  • cyclefridge enable=2, on all rx
  • replaced D low with spare line
  • there is still a leak in compressor 3, but couldn't locate it yet

• 180617 13:27:05 - 180619 08:00:20

  • /home/keck/sch/mce_autotune.sch
  • /data/keckdaq/autoqueue/pending/9_kCMB_50_310.sch
  • mediator stopped
  • restarted gcp/masd
  • 180619 03:44:08 Starting schedule: /home/keck/sch/mce_autotune.sch
  • 180619 03:45:58 Starting schedule: /data/keckdaq/autoqueue/pending/9_kCMB_50_310.sch
  • /home/keck/sch/runend.sch
  • weather: hazy, blowing snow, high winds
  • windows and baffles: windows clear, very little snow in the baffles

• 180617 08:08:13

  • cyclefridge enable=2, on all rx
  • recharged comp 4
  • possible leaks in comp 3 and 4

• 180615 14:08:21 - 180617 08:08:13

  • /home/keck/sch/mce_autotune.sch
  • /data/keckdaq/autoqueue/pending/9_kCMB_50_309.sch
  • /home/keck/sch/runend.sch
  • weather: very hazy, blowing snow, high winds
  • windows and baffles: windows clear, LOTs of snow in the baffles

• 180615 08:16:05

  • cyclefridge enable=2, on all rx
  • replaced bad az line with new spare

• 180613 16:05:07 - 180615 08:16:05

  • /home/keck/sch/mce_autotune.sch
  • /data/keckdaq/autoqueue/pending/9_kCMB_50_308.sch
  • /home/keck/sch/runend.sch
  • weather: hazy
  • windows and baffles: windows clear, little snow in and on the baffles

• 180613 08:23:56

  • cyclefridge enable=2, on all rx
  • replaced big fan in drive amp cabinet
  • el current drifting apart
  • recharged comp 3, 4, 5
  • massive leak in C high, must developed during fridgecycle, psi check showed still normal pressure
  • rx2 is very sensitive to under pressure, even 277psi on the high side made the cycle to fail
  • swapped C high for spare line
  • will miss first 2 scansets

• 180611 14:08:26 - 180613 08:23:56

  • /home/keck/sch/mce_autotune.sch
  • /data/keckdaq/autoqueue/pending/9_kCMB_50_307.sch
  • mediator stopped
  • restarted gcp/masd
  • 180613 04:12:29 Starting schedule: /home/keck/sch/mce_autotune.sch
  • 180613 08:20:36 moving /data/keckdaq/autoqueue/pending/9_kCMB_50_307.sch to /data/keckdaq/autoqueue/done/9_kCMB_50_307.sch
  • /home/keck/sch/runend.sch
  • weather: hazy, blowing snow, wind easing a bit
  • windows and baffles: windows clear, some snow in and on the baffles

• 180611 08:33:00

  • cyclefridge enable=2, on all rx
  • recharged comp 2

• 180609 14:31:56 - 180611 08:33:00

  • /home/keck/sch/mce_autotune.sch
  • /data/keckdaq/autoqueue/pending/9_kCMB_50_306.sch
  • /home/keck/sch/runend.sch
  • weather: hazy, blowing snow
  • windows and baffles: windows clear, some snow in and on the baffles

• 180609 08:39:40

  • cyclefridge enable=2, on all rx

• 180607 14:03:21 - 180609 08:39:40

  • /home/keck/sch/mce_autotune.sch
  • /data/keckdaq/autoqueue/pending/9_kCMB_50_305.sch
  • /home/keck/sch/runend.sch
  • weather: high winds, blowing snow, overcast, hazy - best weather in days
  • windows and baffles: windows clear, quite some snow in and on the baffles

• 180607 08:48:44

  • cyclefridge enable=2, on all rx
  • recharged comp 4

• 180605 14:15:37 - 180607 08:48:44

  • /home/keck/sch/mce_autotune.sch
  • /data/keckdaq/autoqueue/pending/9_kCMB_50_304.sch
  • /home/keck/sch/runend.sch
  • weather: very hazy, blowing snow, high winds, overcast
  • windows and baffles: windows clear, quite some snow in and on the baffles

• 180605 08:55:22

  • cyclefridge enable=2, on all rx

• 180603 14:22:49 - 180605 08:55:22

  • /home/keck/sch/mce_autotune.sch
  • /data/keckdaq/autoqueue/pending/9_kCMB_50_303.sch
  • mediator stopped
  • restarted gcp/masd
  • 180603 21:04:11 Starting schedule: /home/keck/sch/mce_autotune.sch
  • 180603 21:06:00 Starting schedule: /data/keckdaq/autoqueue/pending/9_kCMB_50_303.sch
  • /home/keck/sch/runend.sch
  • weather: haze, blowing snow, high winds
  • windows and baffles: windows clear, quite some snow in and on the baffles

• 180603 09:03:21

  • cyclefridge enable=2, on all rx

• 180601 15:03:23 - 180603 09:03:21

  • /home/keck/sch/mce_autotune.sch
  • /data/keckdaq/autoqueue/pending/9_kCMB_50_302.sch
  • /home/keck/sch/runend.sch
  • weather: haze, high winds, blowing snow, storm, clearing
  • windows and baffles: windows clear, lots of snow in the baffles

• 180601 09:11:09

  • cyclefridge enable=2, on all rx

• 180530 15:11:15 - 180601 09:11:09

  • /home/keck/sch/mce_autotune.sch
  • /data/keckdaq/autoqueue/pending/9_kCMB_50_301.sch
  • mediator stopped
  • restarted gcp/masd
  • 180601 02:22:02 Starting schedule: /home/keck/sch/mce_autotune.sch
  • 180601 02:24:08 Starting schedule: /data/keckdaq/autoqueue/pending/9_kCMB_50_301.sch
  • /home/keck/sch/runend.sch
  • weather: clear, haze, wind picking up, blowing snow
  • windows and baffles: windows clear, some snow in and on the baffles

• 180530 09:27:02

  • cyclefridge enable=2, on all rx
  • recharged comp 2, 3, 4

• 180528 14:45:57 - 180530 09:19:00

  • /home/keck/sch/mce_autotune.sch
  • /data/keckdaq/autoqueue/pending/9_kCMB_50_300.sch
  • mediator stopped
  • restarted gcp/masd
  • 180529 02:11:49 Starting schedule: /home/keck/sch/mce_autotune.sch
  • 180529 02:13:38 Starting schedule: /data/keckdaq/autoqueue/pending/9_kCMB_50_300.sch
  • /home/keck/sch/runend.sch
  • this time rebooting all kecks didn't work and we still had mediator crashes, trying again
  • rebooting all kecks, restarted gcp and masd
  • weather: thin clouds, very hazy, clear
  • windows and baffles: windows clear, some snow in and on the baffles

• 180528 09:26:49

  • cyclefridge enable=2, on all rx

• 180526 14:54:11 - 180528 09:26:49

  • /home/keck/sch/mce_autotune.sch
  • /data/keckdaq/autoqueue/pending/9_kCMB_50_299.sch
  • mediator stopped
  • restarted gcp/masd
  • 180527 02:07:30 Starting schedule: /home/keck/sch/mce_autotune.sch
  • 180527 02:09:19 Starting schedule: /data/keckdaq/autoqueue/pending/9_kCMB_50_299.sch
  • /home/keck/sch/runend.sch
  • weather: mostly clear overhead, wind picking up
  • windows and baffles: clear

• 180526 09:34:43

  • cyclefridge enable=2, on all rx
  • swapped A low for spare line, slow leak

• 180524 15:05:21 - 180526 09:34:43

  • /home/keck/sch/mce_autotune.sch
  • /data/keckdaq/autoqueue/pending/9_kCMB_50_298.sch
  • 180525 19:59 masd4 stopped, restarted keck37 and masd4 came back after a few scansets
  • mediator stopped
  • restarted gcp/masd
  • 180526 03:28:24 Starting schedule: /home/keck/sch/mce_autotune.sch
  • 180526 03:30:13 Starting schedule: /data/keckdaq/autoqueue/pending/9_kCMB_50_298.sch
  • /home/keck/sch/runend.sch
  • weather: clear, mostly clear overhead, wind picking up
  • windows and baffles: clear

• 180524 09:52:00

  • cyclefridge enable=2, on all rx
  • swapped E High for spare line
  • 180524 11:16:47 Starting schedule: /home/keck/sch/starpoint_pole_keck_winter.sch
  • 180524 14:43:36 Exiting schedule: /home/keck/sch/starpoint_pole_keck_winter.sch

• 180522 15:10:31 - 180524 09:42:34

  • /home/keck/sch/mce_autotune.sch
  • /data/keckdaq/autoqueue/pending/9_kCMB_50_297.sch
  • mediator stopped
  • restarted gcp/masd
  • 180523 03:36:28 Starting schedule: /home/keck/sch/mce_autotune.sch
  • 180523 03:38:18 Starting schedule: /data/keckdaq/autoqueue/pending/9_kCMB_50_297.sch
  • mediator stopped
  • restarted gcp/masd
  • 180524 05:31:02 Starting schedule: /home/keck/sch/mce_autotune.sch
  • 180524 05:32:52 Starting schedule: /data/keckdaq/autoqueue/pending/9_kCMB_50_297.sch
  • /home/keck/sch/runend.sch
  • rebooted all keck computers, see if that helps with the mediator crashes
  • weather: blowing snow, very hazy, clearing, clear overhead
  • windows and baffles: windows clear, very little snow in the baffles

• 180522 09:50:28

  • cyclefridge enable=2, on all rx
  • recharged comp 3
  • might be also a small leak

• 180520 15:50:34 - 180522 09:50:28

  • /home/keck/sch/mce_autotune.sch
  • /data/keckdaq/autoqueue/pending/9_kCMB_50_296.sch
  • az diff jump
  • 180520 22:18:46 resume_schedule
  • /home/keck/sch/runend.sch
  • weather: blowing snow, very hazy
  • windows and baffles: windows clear, lots of snow in the baffles

• 180520 09:58:20

  • cyclefridge enable=2, on all rx
  • recharged comp 5
  • slow leak in comp 5
  • el current drifting apart

• 180518 15:58:26 - 180520 09:58:20

  • /home/keck/sch/mce_autotune.sch
  • /data/keckdaq/autoqueue/pending/9_kCMB_50_295.sch
  • mediator stopped
  • restarted gcp/masd
  • 180520 01:33:05 Starting schedule: /home/keck/sch/mce_autotune.sch
  • 180520 01:34:54 Starting schedule: /data/keckdaq/autoqueue/pending/9_kCMB_50_295.sch
  • /home/keck/sch/runend.sch
  • weather: blowing snow, hazy
  • windows and baffles: windows clear, quite a bit of snow on the baffles

• 180518 10:02:50

  • cyclefridge enable=2, on all rx
  • keck32 /data filled up as aftermath from the omega0/data crash
  • even the crontab purge data should only leave 12 days there, for some reason that didn't work and
  • there were still older files filling the disk, which I deleted by hand,
  • since the disk was full we missing part of last run
  • therefore the examiner plots don't show up for this run because it takes the start of the fridgecyle to end it

• 180516 15:39:59 - 180518 10:02:50

  • /home/keck/sch/mce_autotune.sch
  • /data/keckdaq/autoqueue/pending/9_kCMB_50_294.sch
  • mediator stopped
  • restarted gcp/masd
  • 180517 02:49:27 Starting schedule: /home/keck/sch/mce_autotune.sch
  • 180517 02:51:16 Starting schedule: /data/keckdaq/autoqueue/pending/9_kCMB_50_294.sch
  • /home/keck/sch/runend.sch
  • weather: clearing, not for long :(, blowing snow, warm overcast, haze
  • windows and baffles: windows clear, quite some snow in and on the baffles

• 180516 10:14:03

  • cyclefridge enable=2, on all rx
  • recharged comp 4

• 180514 16:14:10 - 180516 10:14:03

  • /home/keck/sch/mce_autotune.sch
  • /data/keckdaq/autoqueue/pending/9_kCMB_50_293.sch
  • mediator stopped
  • restarted gcp/masd
  • 180516 01:14:36 Starting schedule: /home/keck/sch/mce_autotune.sch
  • 180516 01:16:28 Starting schedule: /data/keckdaq/autoqueue/pending/9_kCMB_50_293.sch
  • /home/keck/sch/runend.sch
  • weather: warm, blowing snow, haze, overcast, clearing to the end
  • windows and baffles: windows clear, some snow in and on the baffles

• 180514 10:21:53

  • cyclefridge enable=2, on all rx
  • recharged comp 2, 5

• 180512 16:22:02 - 180514 10:21:53

  • /home/keck/sch/mce_autotune.sch
  • /data/keckdaq/autoqueue/pending/9_kCMB_50_292.sch
  • /home/keck/sch/runend.sch
  • weather: haze and blowing snow, very warm, totally overcast
  • windows and baffles: windows clear, some snow in and on the baffles

• 180512 10:29:47

  • cyclefridge enable=2, on all rx
  • recharged comp 3

• 180510 16:04:08 - 180512 10:29:47

  • /home/keck/sch/mce_autotune.sch
  • /data/keckdaq/autoqueue/pending/9_kCMB_50_291.sch
  • mediator stopped
  • restarted gcp/masd
  • 180511 03:03:05 Starting schedule: /home/keck/sch/mce_autotune.sch
  • 180511 03:04:55 Starting schedule: /data/keckdaq/autoqueue/pending/9_kCMB_50_291.sch
  • /home/keck/sch/runend.sch
  • weather: haze and blowing snow, very warm
  • windows and baffles: windows clear, little bit of snow in the baffles

• 180510 10:37:18

  • cyclefridge enable=2, on all rx
  • omega0/data working again, all data caught up
  • examiner plots working again as well

• 180508 16:37:45 - 180510 10:37:18

  • /home/keck/sch/mce_autotune.sch
  • /data/keckdaq/autoqueue/pending/9_kCMB_50_290.sch
  • rx0 FPU temp went up at runstart to the normal higher temperature BUT stayed up
  • not the normal drop quickly back below 300mK, it is cooling down slowly
  • rx0 staying above 300mK strange behavior, since UC is cold !??
  • new /data on omega0 is in place, files getting transfered to spfs
  • rx0 FPU still around 326mK, tried again to restart keck33 and masd0
  • /home/keck/sch/runend.sch
  • weather: hazy, blowing snow, clearing, hazy, blowing snow again, warm
  • windows and baffles: windows clear, some snow in and on the baffles

• 180508 10:49:49

  • cyclefridge enable=2, on all rx
  • restarted keck33 and masd0

• 180506 16:45:37 - 180508 10:49:49

  • /home/keck/sch/mce_autotune.sch
  • /data/keckdaq/autoqueue/pending/9_kCMB_50_289.sch
  • masd0 stopped at the end of the run
  • /home/keck/sch/runend.sch
  • weather: hazy, blowing snow
  • windows and baffles: windows clear some snow in and on the baffles

• 180506 10:53:28

  • cyclefridge enable=2, on all rx
  • recharged comp 5

• 180504 16:53:29 - 180506 10:53:28

  • /home/keck/sch/mce_autotune.sch
  • /data/keckdaq/autoqueue/pending/9_kCMB_50_288.sch
  • problems with omega0 disk array, Clem got it running for a couple of hours
  • but than it died again
  • therefore no data, examiner etc.
  • /home/keck/sch/runend.sch
  • weather: little hazy, blowing snow, very hazy
  • windows and baffles: windows clear, some snow in and on the baffles

• 180504 11:01:12

  • cyclefridge enable=2, on all rx

• 180502 17:01:21 - 180504 11:01:12

  • /home/keck/sch/mce_autotune.sch
  • /data/keckdaq/autoqueue/pending/9_kCMB_50_287.sch
  • /home/keck/sch/runend.sch
  • weather: clear overhead, cold, warming up, haze, blowing snow
  • windows and baffles: windows clear, some snow in the baffles

• 180502 11:09:04

  • cyclefridge enable=2, on all rx
  • el currents drifting apart

• 180430 17:34:23 - 180502 11:09:04

  • /home/keck/sch/mce_autotune.sch
  • /data/keckdaq/autoqueue/pending/9_kCMB_50_286.sch
  • /home/keck/sch/runend.sch
  • weather: mostly clear, cold
  • windows and baffles: windows clear, very little snow in the baffles

• 180430 11:16:56

  • cyclefridge enable=2, on all rx
  • swapped bad line for new spare
  • recharged comp 2
  • problems with pamc handcontroller, ribbon cable came lose
  • shutdown keck31 to work on it, therefore had to restart fridgecycle
  • 180430 12:14:19 Starting schedule: /home/keck/sch/fridgecycle.sch

• 180428 16:49:01 - 180430 11:16:56

  • /home/keck/sch/mce_autotune.sch
  • /data/keckdaq/autoqueue/pending/9_kCMB_50_285.sch
  • /home/keck/sch/runend.sch
  • weather: clear, mostly clear
  • windows and baffles: clear

• 180428 11:24:39

  • cyclefridge enable=2, on all rx

• 180426 17:24:56 - 180428 11:24:39

  • /home/keck/sch/mce_autotune.sch
  • /data/keckdaq/autoqueue/pending/9_kCMB_50_284.sch
  • /home/keck/sch/runend.sch
  • weather: hazy, claer
  • windows and baffles: clear

• 180426 11:32:30

  • cyclefridge enable=2, on all rx
  • recharged comp 3, 4
  • possible leak comp 3
  • swapped A high for spare line

• 180424 17:32:48 - 180426 11:32:30

  • /home/keck/sch/mce_autotune.sch
  • /data/keckdaq/autoqueue/pending/9_kCMB_50_283.sch
  • /home/keck/sch/runend.sch
  • weather: partly cloudy, haze
  • windows and baffles: windows clear, little snow in and on the baffles

• 180424 11:40:22

  • cyclefridge enable=2, on all rx
  • recharged comp 5

• 180422 17:48:14 - 180424 11:40:22

  • /home/keck/sch/mce_autotune.sch
  • /data/keckdaq/autoqueue/pending/9_kCMB_50_282.sch
  • aborted run, new bias not loaded yet
  • 180422 17:55:38 Starting schedule: /home/keck/sch/mce_autotune.sch
  • 180422 17:57:28 Starting schedule: /data/keckdaq/autoqueue/pending/9_kCMB_50_282.sch
  • new bias for rx4
  • /home/keck/sch/runend.sch
  • weather: clear, partly cloudy, haze
  • windows and baffles: windows clear, some snow in and on the baffles

• 180422 12:01:15

  • cyclefridge enable=2, on all rx
  • 180422 14:09:15 Starting schedule: /home/keck/sch/starpoint_pole_keck_winter.sch
  • 180422 17:48:14 Exiting schedule: /home/keck/sch/starpoint_pole_keck_winter.sch

• 180420 17:16:56 - 180422 11:48:15

  • /home/keck/sch/mce_autotune.sch
  • /data/keckdaq/autoqueue/pending/9_kCMB_50_281.sch
  • /home/keck/sch/runend.sch
  • weather: thins clouds, blowing snow, clearing
  • windows and baffles: windows clear, little snow in the baffles

• 180420 11:56:07

  • cyclefridge enable=2, on all rx

• 180418 17:18:09 - 180420 11:56:07

  • /home/keck/sch/mce_autotune.sch
  • /data/keckdaq/autoqueue/pending/9_kCMB_50_280.sch
  • /home/keck/sch/runend.sch
  • weather: clear, thin clouds, haze, blowing snow
  • windows and baffles: windows clear, quite some snow in and on the baffles

• 180418 12:03:59

  • cyclefridge enable=2, on all rx

• 180416 18:20:45 - 180418 12:03:59

  • /home/keck/sch/mce_autotune.sch
  • /data/keckdaq/autoqueue/pending/9_kCMB_50_279.sch
  • /home/keck/sch/runend.sch
  • weather: clear, overcast, clearing
  • windows and baffles: windows clear, some snow in and on the baffles

• 180416 13:05:06

  • cyclefridge enable=2, on all rx

• 180414 17:44:24 - 180416 12:11:52

  • /home/keck/sch/mce_autotune.sch
  • /data/keckdaq/autoqueue/pending/9_kCMB_50_278.sch
  • running without rx1
  • 180415 03:00 rx1 is cold again and will be included
  • mediator crashed
  • restarted gcp/masd
  • this time I actually saw the crash live and I new it would crash, values alternated in quick succession with actual values
  • and "0" for some values, pmac would show "running", "stopped", "running"... the rate was about 4Hz
  • 180415 03:17:19 schedule /home/keck/sch/mce_autotune.sch
  • 180415 03:21:00 Starting schedule: /data/keckdaq/autoqueue/pending/9_kCMB_50_278.sch
  • /home/keck/sch/runend.sch
  • restarted keck32, to see if that fixes pager problem
  • also rebooted keck31 and the other keck computers
  • diskcheck after reboot on keck32 and keck31 nearly took one hour
  • restarted gcp/masd
  • weather: overcast, wind picking up, clearing, clear
  • windows and baffles: windows clear, little snow in the baffles

• 180414 12:19:45

  • cyclefridge enable=2, on all rx
  • massive He leak on rx1, fridgecycle didn't finish, changed to spare line

• 180412 17:54:41 - 180414 12:19:45

  • /home/keck/sch/mce_autotune.sch
  • /data/keckdaq/autoqueue/pending/9_kCMB_50_277.sch
  • mediator crashed
  • restarted gcp/masd
  • 180413 01:05:00 Starting schedule: /home/keck/sch/mce_autotune.sch
  • 180413 01:06:50 Starting schedule: /data/keckdaq/autoqueue/pending/9_kCMB_50_277.sch
  • /home/keck/sch/runend.sch
  • weather: clear, getting overcast, wind picking up
  • windows and baffles: windows clear, some snow on and in the baffles

• 180412 12:27:36

  • cyclefridge enable=2, on all rx

• 180410 18:27:51 - 180412 12:27:36

  • /home/keck/sch/mce_autotune.sch
  • /data/keckdaq/autoqueue/pending/9_kCMB_50_276.sch
  • my screw up, didn't put new schedule in pending
  • 180411 00:05:20 Starting schedule: /data/keckdaq/autoqueue/pending/9_kCMB_50_276.sch
  • but pager didn't go off either, experimenting with pager to see how that can be prevented
  • /home/keck/sch/runend.sch
  • weather: mostly clear, clear, getting cold
  • windows and baffles: clear

• 180410 12:35:27

  • cyclefridge enable=2, on all rx
  • recharged comp 3

• 180408 18:35:43 - 180410 12:35:27

  • /home/keck/sch/mce_autotune.sch
  • /data/keckdaq/autoqueue/pending/9_kCMB_50_275.sch
  • mediator crashed
  • restarted gcp/masd
  • 180409 10:41:52 Starting schedule: /home/keck/sch/mce_autotune.sch
  • 180409 10:43:43 Starting schedule: /data/keckdaq/autoqueue/pending/9_kCMB_50_275.sch
  • /home/keck/sch/runend.sch
  • weather: mostly clear
  • windows and baffles: clear

• 180408 12:43:18

  • cyclefridge enable=2, on all rx
  • 180408 15:10:29 Starting schedule: /home/keck/sch/starpoint_pole_keck_summer.sch
  • 180408 17:26:57 Exiting schedule: /home/keck/sch/starpoint_pole_keck_summer.sch

• 180406 18:25:38 - 180408 12:43:18

  • /home/keck/sch/mce_autotune.sch
  • /data/keckdaq/autoqueue/pending/9_kCMB_50_274.sch
  • mediator crashed
  • restarted gcp/masd
  • 180407 03:19:40 Starting schedule: /home/keck/sch/mce_autotune.sch
  • 180407 03:21:31 Starting schedule: /data/keckdaq/autoqueue/pending/9_kCMB_50_274.sch
  • /home/keck/sch/runend.sch
  • weather: overcast, clearing, clear
  • windows and baffles: clear

• 180406 12:59:08

  • cyclefridge enable=2, on all rx

• 180404 18:17:15 - 180406 12:51:12

  • /home/keck/sch/mce_autotune.sch
  • /data/keckdaq/autoqueue/pending/9_kCMB_50_273.sch
  • /home/keck/sch/runend.sch
  • restarted gcp/masd, pointing prolbem is fixed
  • weather: thin clouds, overcast
  • windows and baffles: very little snow on the windows, some snow in and on the baffles

• 180404 13:02:28

  • cyclefridge enable=2, on all rx
  • recharged comp 2,4

• 180402 18:59:18 - 180404 12:59:03

  • /home/keck/sch/mce_autotune.sch
  • /data/keckdaq/autoqueue/pending/9_kCMB_50_272.sch
  • 180404 04:00 wifi and bluetooth on singnalhound laptop disabled
  • /home/keck/sch/runend.sch
  • weather: clear, clouds moving in
  • windows and baffles: clear

• 180402 13:07:02

  • cyclefridge enable=2, on all rx
  • tried star pointing, but telescope won't slew to correct coordinates??

• 180331 19:07:10 - 180402 13:07:02

  • /home/keck/sch/mce_autotune.sch
  • /data/keckdaq/autoqueue/pending/9_kCMB_50_271.sch
  • /home/keck/sch/runend.sch
  • weather: clear
  • windows and baffles: clear

• 180331 13:36:50

  • cyclefridge enable=2, on all rx
  • recharged comp 5

• 180329 19:15:02 - 180331 13:11:34

  • /home/keck/sch/mce_autotune.sch
  • /data/keckdaq/autoqueue/pending/9_kCMB_50_270.sch
  • /home/keck/sch/runend.sch
  • restarted gcp/masd to get ephem working properly
  • weather: mostly clear, clear
  • windows and baffles: clear

• 180329 13:22:37

  • cyclefridge enable=2, on all rx

• 180327 19:22:53 - 180329 13:22:37

  • /home/keck/sch/mce_autotune.sch
  • /data/keckdaq/autoqueue/pending/9_kCMB_50_269.sch
  • /home/keck/sch/runend.sch
  • weather: thin clouds
  • windows and baffles: windows clear, very little snow in the baffles

• 180327 13:41:10

  • cyclefridge enable=2, on all rx

• 180325 19:30:45 - 180327 13:33:50

  • /home/keck/sch/mce_autotune.sch
  • /data/keckdaq/autoqueue/pending/9_kCMB_50_268.sch
  • /home/keck/sch/runend.sch
  • stopped gcp/masd
  • reverted to gcp version from yr7
  • restarted gcp/masd
  • weather: mostly clear overhead, some thin clouds
  • windows and baffles: clear

• 180325 13:41:48

  • cyclefridge enable=2, on all rx

• 180323 19:38:37 - 180325 13:41:48

  • /home/keck/sch/mce_autotune.sch
  • /data/keckdaq/autoqueue/pending/9_kCMB_50_267.sch
  • 180324 01:36:42 abort_schedule
  • to commit change of gcp back to 2017 version, weather is bad anyway
  • controlSystem won't start properly, going back to year8 version
  • 180324 02:40:34 Starting schedule: /home/keck/sch/mce_autotune.sch
  • 180324 02:42:24 Starting schedule: /data/keckdaq/autoqueue/pending/9_kCMB_50_267.sch
  • /home/keck/sch/runend.sch
  • weather: overcast, high winds, blowing snow, clearing, clear
  • windows and baffles: clear

• 180323 13:49:34

  • cyclefridge enable=2, on all rx
  • recharged comp 2,3

• 180321 19:10:58 - 180323 13:49:34

  • /home/keck/sch/mce_autotune.sch
  • /data/keckdaq/autoqueue/pending/9_kCMB_50_266.sch
  • gcp stopped, mediator ?? different from last years mediator crashes
  • restarted gcp/masd
  • 180322 02:31:13 Starting schedule: /home/keck/sch/mce_autotune.sch
  • 180322 02:33:02 Starting schedule: /data/keckdaq/autoqueue/pending/9_kCMB_50_266.sch
  • /home/keck/sch/runend.sch
  • weather: mostly clear overhead, overcast, windy, blowing snow
  • windows and baffles: windows clear, some snow in the baffles

• 180321 13:53:30

  • cyclefridge enable=2, on all rx

• 180319 19:54:21 - 180321 13:53:30

  • /home/keck/sch/mce_autotune.sch
  • /data/keckdaq/autoqueue/pending/9_kCMB_50_265.sch
  • 180319 22:35 brief power outage
  • el current drifting apart
  • 180320 14:15:15 Exiting schedule: /data/keckdaq/autoqueue/pending/9_kCMB_50_265.sch ??????????????????
  • restarted gcp/masd, otherwise keckviewer wasn't running properly
  • rx1 warming up slightly
  • 180320 14:39:21 Starting schedule: /home/keck/sch/mce_autotune.sch
  • 180320 14:41:11 Starting schedule: /data/keckdaq/autoqueue/pending/9_kCMB_50_265.sch
  • mediator stopped, different from normal
  • restarted gcp/masd
  • 180321 02:40:05 Starting schedule: /data/keckdaq/autoqueue/pending/9_kCMB_50_265.sch
  • /home/keck/sch/runend1.sch
  • weather: mostly clear overhead, wind picking up
  • windows and baffles: windows clear, very little snow in the baffles

• 180319 14:05:16

  • cyclefridge enable=2, on all rx
  • 180319 15:38:06 Starting schedule: /home/keck/sch/starpoint_pole_keck_summer.sch
  • 180319 17:58:27 Exiting schedule: /home/keck/sch/starpoint_pole_keck_summer.sch

• 180317 20:02:13 - 180319 14:05:16

  • /home/keck/sch/mce_autotune.sch
  • /data/keckdaq/autoqueue/pending/9_kCMB_50_264.sch
  • mediator stopped
  • restarted gcp/masd
  • 180319 02:00:59 Starting schedule: /home/keck/sch/mce_autotune.sch
  • 180319 02:02:48 Starting schedule: /data/keckdaq/autoqueue/pending/9_kCMB_50_264.sch
  • /home/keck/sch/runend.sch
  • weather: totally overcast, windy, clearing
  • windows and baffles: windows clear, some snow in the baffles

• 180317 14:13:09

  • cyclefridge enable=2, on all rx
  • el currents drifting apart
  • bicep spare variac had reversed polarity, therefore breaker triped, used it now for spares
  • for our variac

• 180315 19:40:05 - 180317 14:13:09

  • /home/keck/sch/mce_autotune.sch
  • /data/keckdaq/autoqueue/pending/1_knoise_responsivity_9_024.sch
  • el currents drifting apart
  • 180316 01:57:35 Starting schedule: /data/keckdaq/autoqueue/pending/9_kCMB_50_263.sch
  • /home/keck/sch/runend.sch
  • weather: clear, windy, totally overcast
  • windows and baffles: very little snow on the windows, lots of snow in and on the baffles

• 180315 14:21:01

  • cyclefridge enable=2, on all rx
  • installed new spare az line
  • el currents drifting apart
  • disconnected all baffle temperature sensors, far fetched idea, but these are the only sensors that are
  • not used during calibration when we don't have frequent mediator crashes

• 180313 20:18:56 - 180315 14:21:01

  • /home/keck/sch/mce_autotune.sch
  • /data/keckdaq/autoqueue/pending/9_kCMB_50_262.sch
  • run stopped ?? pager couldn't be turned off until restart of gcp, masd
  • keckviewer was still in downsample 1
  • 180314 07:33:47 Starting schedule: /home/keck/sch/mce_autotune.sch
  • 180314 07:35:46 Starting schedule: /data/keckdaq/autoqueue/pending/9_kCMB_50_262.sch
  • mediator crashed
  • 180315 08:59:17 abort_schedule
  • restarted gcp/masd
  • 180315 09:01:13 Starting schedule: /home/keck/sch/mce_autotune.sch
  • 180315 09:03:16 Starting schedule: /data/keckdaq/autoqueue/pending/9_kCMB_50_262.sch
  • /home/keck/sch/runend.sch
  • weather: mostly clear
  • one variac broke, carbon brush broke off
  • windows and baffles: windows clear, little snow in the baffles

• 180313 14:25:00

  • cyclefridge enable=2, on all rx
  • 180313 17:19:21 Starting schedule: /home/keck/sch/starpoint_pole_keck_summer.sch
  • missing too many stars, aborted star pointing

• 180311 20:25:48 - 180313 14:25:00

  • /home/keck/sch/mce_autotune.sch
  • /data/keckdaq/autoqueue/pending/9_kCMB_50_261.sch
  • He leak in D high line, rx3 warmed up
  • accidently stopped gcp, restarted gcp and masd
  • 180312 18:43:16 Starting schedule: /home/keck/sch/mce_autotune.sch
  • 180312 18:45:06 Starting schedule: /data/keckdaq/autoqueue/pending/9_kCMB_50_261.sch
  • swapped D high for spare line
  • /home/keck/sch/runend3.sch
  • weather: overcast, blowing snow, clearing, mostly clear
  • windows and baffles: windows clear, very little snow in the baffles

• 180311 14:36:44

  • cyclefridge enable=2, on all rx

• 180309 20:33:40 - 180311 14:36:44

  • /home/keck/sch/mce_autotune.sch
  • /data/keckdaq/autoqueue/pending/9_kCMB_50_260.sch
  • mediator crashed :( no pager alarm
  • 180310 02:05:56 abort_schedule
  • restarted gcp and masd
  • 180310 02:07:32 Starting schedule: /home/keck/sch/mce_autotune.sch
  • 180310 02:09:27 Starting schedule: /data/keckdaq/autoqueue/pending/9_kCMB_50_260.sch
  • /home/keck/sch/runend.sch
  • weather: clearing, overcast, blowing snow
  • windows and baffles: windows clear, some snow in and on the baffles

• 180309 14:44:35

  • cyclefridge enable=2, on all rx
  • 180309 15:30:44 Starting schedule: /home/keck/sch/starpoint_pole_keck_summer.sch
  • aborted star pointing, missing too many stars

• 180307 20:41:32 - 180309 14:44:35

  • /home/keck/sch/mce_autotune.sch
  • /data/keckdaq/autoqueue/pending/9_kCMB_50_259.sch
  • keckViewer was unresponsive, didn't look like the normal mediator crash AND and I got a page
  • restarted gcp and masd
  • 180308 17:33:32 Starting schedule: /home/keck/sch/mce_autotune.sch
  • 180308 17:35:21 Starting schedule: /data/keckdaq/autoqueue/pending/9_kCMB_50_259.sch
  • /home/keck/sch/runend.sch
  • weather: overcast, blowing snow, clearing
  • windows and baffles: windows clear, some snow in and on the baffles

• 180307 11:22:39

  • removed mirror frame and installed forebaffles
  • getting telescope ready for first cmb
  • recharged comp 2
  • cyclefridge enable=2, on all rx
  • keckviewer showed "0" on all entries after starting fridgecycle??? mediator was running
  • restarted gcp and masd
  • 180307 11:25:34 Starting schedule: /home/keck/sch/fridgecycle.sch

---

CMB 2018

---

• 180306 08:03:17

  • cyclefridge enable=2, on all rx
  • tried star pointing, but thin clouds

• 180304 19:59:18

  • cyclefridge enable=2, on all rx
  • mediator crashed, after the cycle, didn't restarted until I was back out at MAPO, since the telescope was idle anyway
  • restarted gcp and masd
  • 180306 04:45 mirror is removed, telescope is cooling back down, all well
  • warm up of FPUs from 12K to 19K

• 180304 12:17:42 - 180304 19:59:18

  • /home/keck/sch/mce_autotune.sch
  • /data/keckdaq/autoqueue/pending/7_ffflat_dslmast_33j_014.sch
  • weather: thin clouds

• 180304 07:10:26

  • cyclefridge enable=2, on all rx

• 180303 23:28:09 - 180304 07:10:26

  • /home/keck/sch/mce_autotune.sch
  • /data/keckdaq/autoqueue/pending/7_ffflat_dslmast_33i_014.sch
  • weather: thin clouds

• 180303 18:24:39

  • cyclefridge enable=2, on all rx

• 180303 10:41:04 - 180303 18:24:39

  • /home/keck/sch/mce_autotune.sch
  • /data/keckdaq/autoqueue/pending/7_ffflat_dslmast_33h_014.sch
  • weather: thin clouds

• 180303 05:38:42

  • cyclefridge enable=2, on all rx

• 180302 21:56:47 - 180303 05:38:42

  • /home/keck/sch/mce_autotune.sch
  • /data/keckdaq/autoqueue/pending/7_ffflat_dslmast_33g_014.sch
  • /home/keck/sch/runend.sch
  • weather: thin clouds

• 180302 14:11:32

  • cyclefridge enable=2, on all rx

• 180302 06:29:06 - 180302 14:11:32

  • /home/keck/sch/mce_autotune.sch
  • /data/keckdaq/autoqueue/pending/7_ffflat_dslmast_33f_014.sch
  • weather: slightly overcast

• 180302 01:11:39

  • cyclefridge enable=2, on all rx

• 180301 17:30:00 - 180302 01:11:39

  • /home/keck/sch/mce_autotune.sch
  • /data/keckdaq/autoqueue/pending/7_ffflat_dslmast_33e_014.sch
  • weather: overcast

• 180301 11:45:37

  • cyclefridge enable=2, on all rx

• 180301 04:02:20 - 180301 11:45:37

  • /home/keck/sch/mce_autotune.sch
  • /data/keckdaq/autoqueue/pending/7_ffflat_dslmast_33d_014.sch
  • weather: clear

• 180228 22:42:20

  • cyclefridge enable=2, on all rx

• 180228 15:00:00 - 180228 22:42:19

  • /home/keck/sch/mce_autotune.sch
  • /data/keckdaq/autoqueue/pending/7_ffflat_dslmast_33c_014.sch
  • weather: clear

• 180228 08:59:10

  • cyclefridge enable=2, on all rx

• 180228 01:16:27 - 180228 08:59:10

  • /home/keck/sch/mce_autotune.sch
  • /data/keckdaq/autoqueue/pending/7_ffflat_dslmast_33b_014.sch
  • weather: clear

• 180227 20:15:56

  • cyclefridge enable=2, on all rx

• 180227 12:34:38 - 180227 20:15:55

  • /home/keck/sch/mce_autotune.sch
  • /data/keckdaq/autoqueue/pending/7_ffflat_dslmast_33a_014.sch
  • weather: clear

• 180227 07:33:49

  • cyclefridge enable=2, on all rx

• 180226 23:52:15 - 180227 07:33:48

  • /home/keck/sch/mce_autotune.sch
  • /data/keckdaq/autoqueue/pending/7_ffflat_dslmast_33j_013.sch
  • weather: clear

• 180226 18:29:05

  • cyclefridge enable=2, on all rx

• 180226 10:46:58 - 180226 18:29:04

  • /home/keck/sch/mce_autotune.sch
  • /data/keckdaq/autoqueue/pending/7_ffflat_dslmast_33i_013.sch
  • weather: clear

• 180226 05:51:09

  • cyclefridge enable=2, on all rx

• 180225 22:07:53 - 180226 05:51:09

  • /home/keck/sch/mce_autotune.sch
  • /data/keckdaq/autoqueue/pending/7_ffflat_dslmast_33h_013.sch
  • weather: clear

• 180225 15:46:38

  • cyclefridge enable=2, on all rx

• 180225 08:04:41 - 180225 15:46:38

  • /home/keck/sch/mce_autotune.sch
  • /data/keckdaq/autoqueue/pending/7_ffflat_dslmast_33g_013.sch
  • weather: clear, blowing snow

• 180225 03:04:14

  • cyclefridge enable=2, on all rx

• 180224 19:21:56 - 180225 03:04:14

  • /home/keck/sch/mce_autotune.sch
  • /data/keckdaq/autoqueue/pending/7_ffflat_dslmast_33f_013.sch
  • weather: clear, blowing snow

• 180224 11:05:58

  • cyclefridge enable=2, on all rx

• 180224 03:24:07 - 180224 11:05:57

  • /home/keck/sch/mce_autotune.sch
  • /data/keckdaq/autoqueue/pending/7_ffflat_dslmast_33e_013.sch
  • weather: clear

• 180223 22:08:13

  • cyclefridge enable=2, on all rx

• 180223 14:23:39 - 180223 22:08:13

  • /home/keck/sch/mce_autotune.sch
  • /data/keckdaq/autoqueue/pending/7_ffflat_dslmast_33d_013.sch
  • weather: clear

• 180223 05:48:01

  • cyclefridge enable=2, on all rx

• 180222 22:06:27 - 180223 05:48:01

  • /home/keck/sch/mce_autotune.sch
  • /data/keckdaq/autoqueue/pending/7_ffflat_dslmast_33c_013.sch
  • weather: clear

• 180221 22:55:48

  • cyclefridge enable=2, on all rx

• 180221 15:13:18 - 180221 22:55:47

  • /home/keck/sch/mce_autotune.sch
  • /data/keckdaq/autoqueue/pending/7_ffflat_dslmast_33b_013.sch
  • weather: clear

• 180221 09:32:23

  • cyclefridge enable=2, on all rx

• 180221 01:12:02 - 180221 09:32:23

  • /home/keck/sch/mce_autotune.sch
  • /data/keckdaq/autoqueue/pending/7_ffflat_dslmast_33a_013.sch
  • 180221 01:37:00 abort_schedule
  • did a sign mistake while adjusting the front bottom ones, we went to 25.75" instead of 21"
  • so aborted the run and corrected this
  • 180221 01:53:02 Starting schedule: /data/keckdaq/autoqueue/pending/7_ffflat_dslmast_33a_013.sch
  • weather: clear

• 180220 16:00:20

  • cyclefridge enable=2, on all rx
  • moved mirror to the back postion
  • becaus of block mix up we meassured the current values
  • front bottom: should have been at 15" was at 12.75" so -2.25"
  • front top: should have been at 55.5" was at 52" (both were about 1" different 52 is the middle value) so -3.5"
  • back: should have been at 29" was at 30.75" so + 1.75"
  • after adjustment to "back" postion
  • front bottom: should have been at 23.5" -2.25" so is now at 21.25" (originally did a mistake here, see above)
  • front top: should have been at 57.5" -3.5" so is now at 54"
  • back: should have been at 25.2" +1.75 so is now at 27"

• 180220 08:18:23 - 180220 16:00:19

  • /home/keck/sch/mce_autotune.sch
  • /data/keckdaq/autoqueue/pending/7_ffflat_dslmast_33j_012.sch
  • weather: clear

• 180220 02:35:38

  • cyclefridge enable=2, on all rx
  • recharged comp2

• 180219 18:53:28 - 180220 02:35:37

  • /home/keck/sch/mce_autotune.sch
  • /data/keckdaq/autoqueue/pending/7_ffflat_dslmast_33i_012.sch
  • weather: clear

• 180219 10:55:36

  • cyclefridge enable=2, on all rx

• 180219 03:12:21 - 180219 10:55:36

  • /home/keck/sch/mce_autotune.sch
  • /data/keckdaq/autoqueue/pending/7_ffflat_dslmast_33h_012.sch
  • weather: clear

• 180218 22:06:00

  • cyclefridge enable=2, on all rx

• 180218 14:24:08 - 180218 22:06:00

  • /home/keck/sch/mce_autotune.sch
  • /data/keckdaq/autoqueue/pending/7_ffflat_dslmast_33g_012.sch
  • weather: clear

• 180218 09:21:39

  • cyclefridge enable=2, on all rx

• 180218 01:37:33 - 180218 09:21:39

  • /home/keck/sch/mce_autotune.sch
  • /data/keckdaq/autoqueue/pending/7_ffflat_dslmast_33f_012.sch
  • weather: clear

• 180217 20:31:09

  • cyclefridge enable=2, on all rx

• 180217 12:48:31 - 180217 20:31:09

  • /home/keck/sch/mce_autotune.sch
  • /data/keckdaq/autoqueue/pending/7_ffflat_dslmast_33e_012.sch
  • weather: clearing, wind easing

• 180217 07:40:16

  • cyclefridge enable=2, on all rx

• 180216 23:57:00 - 180217 07:40:16

  • /home/keck/sch/mce_autotune.sch
  • /data/keckdaq/autoqueue/pending/7_ffflat_dslmast_33d_012.sch
  • weather: overcast, blowing snow

• 180216 18:22:47

  • cyclefridge enable=2, on all rx

• 180216 10:41:13 - 180216 18:22:47

  • /home/keck/sch/mce_autotune.sch
  • /data/keckdaq/autoqueue/pending/7_ffflat_dslmast_33c_012.sch
  • weather: clearing, getting overcast again, wind picking up

• 180216 05:18:56

  • cyclefridge enable=2, on all rx

• 180215 21:36:58 - 180216 05:18:56

  • /home/keck/sch/mce_autotune.sch
  • /data/keckdaq/autoqueue/pending/7_ffflat_dslmast_33b_012.sch
  • weather: overcast, blowing snow

---

Winter 2018

---

• 180215 16:15:11

  • cyclefridge enable=2, on all rx

• 180215 08:33:30 - 180215 16:15:11

  • /home/keck/sch/mce_autotune.sch
  • /data/keckdaq/autoqueue/pending/7_ffflat_dslmast_33a_012.sch
  • weather: overcast, blowing snow
  • windows and baffles:

• 180215 03:21:55

  • cyclefridge enable=2, on all rx

• 180214 19:40:22 - 180215 03:21:55

  • /home/keck/sch/mce_autotune.sch
  • /data/keckdaq/autoqueue/pending/7_ffflat_dslmast_33j_011.sch
  • weather: overcast, blowing snow

• 180214 10:14:11

  • cyclefridge enable=2, on all rx

• 180214 02:32:08 - 180214 10:14:11

  • /home/keck/sch/mce_autotune.sch
  • /data/keckdaq/autoqueue/pending/7_ffflat_dslmast_33i_011.sch
  • weather: fine up to now, but winds picking up

• 180213 21:13:15

  • cyclefridge enable=2, on all rx

• 180213 13:30:00 - 180213 21:13:15

  • /home/keck/sch/mce_autotune.sch
  • /data/keckdaq/autoqueue/pending/7_ffflat_dslmast_33h_011.sch
  • weather: fine, mostly clear skies

• 180213 07:54:27

  • cyclefridge enable=2, on all rx

• 180213 00:12:03 - 180213 07:54:27

  • /home/keck/sch/mce_autotune.sch
  • /data/keckdaq/autoqueue/pending/7_ffflat_dslmast_33g_011.sch
  • weather: fine, mostly clear skies

• 180212 17:24:57

  • cyclefridge enable=2, on all rx

• 180212 09:43:05 - 180212 17:24:56

  • /home/keck/sch/mce_autotune.sch
  • /data/keckdaq/autoqueue/pending/7_ffflat_dslmast_33f_011.sch
  • weather: fine, mostly clear skies

• 180212 04:19:06

  • cyclefridge enable=2, on all rx

• 180211 20:36:29 - 180212 04:19:05

  • /home/keck/sch/mce_autotune.sch
  • /data/keckdaq/autoqueue/pending/7_ffflat_dslmast_33e_011.sch
  • weather: fine, mostly clear skies

• 180211 09:41:07

  • cyclefridge enable=2, on all rx

• 180211 01:57:05 - 180211 09:41:07

  • /home/keck/sch/mce_autotune.sch
  • /data/keckdaq/autoqueue/pending/7_ffflat_dslmast_33d_011.sch

• 180210 07:07:21

  • cyclefridge enable=2, on all rx
  • pmac stopped

• 180209 23:25:48 - 180210 07:07:20

  • /home/keck/sch/mce_autotune.sch
  • data/keckdaq/autoqueue/pending/7_ffflat_dslmast_33c_011.sch

• 180209 14:15:05

  • cyclefridge enable=2, on all rx

• 180209 06:32:15 - 180209 14:15:05

  • /home/keck/sch/mce_autotune.sch
  • /data/keckdaq/autoqueue/pending/7_ffflat_dslmast_33b_011.sch

• 180208 05:41:59

  • cyclefridge enable=2, on all rx

• 180207 22:01:38 - 180208 05:41:59

  • /home/keck/sch/mce_autotune.sch
  • /data/keckdaq/autoqueue/pending/7_ffflat_dslmast_33a_011.sch
  • repeat the previous schedule.
  • forwardest mirror position. (for real this time)
  • we adjust the CF legs to account for the clamp block swaps.
  • front top goes up by 1.75 inch and front bottom goes down by 1.75 inch.
  • chopper ref connected to optical fiber

• 180207 03:37:52 - 180207 11:18:33

  • /home/keck/sch/mce_autotune.sch
  • /data/keckdaq/autoqueue/pending/7_ffflat_dslmast_33a_010.sch
  • forwardest mirror position.
  • DSL chopper up. First FFBM with 16Hz chopper.
  • Forgot to plug in optical chop ref, so will have to redo this schedule.
  • El motor current diverges.

• 180206 06:16:02 - 180206 13:57:03

  • /home/keck/sch/mce_autotune.sch
  • /data/keckdaq/autoqueue/pending/7_ffflat_dslmast_33e_010.sch
  • repeat the schedule with the mediator crash.
  • after this we'll put the chopper up DSL mast and start regular FFBM.
  • forwardest mirror position.

• 180205 16:19:24 - 180205 23:59:09

  • /home/keck/sch/mce_autotune.sch
  • /data/keckdaq/autoqueue/pending/7_ffflat_dslmast_33j_010.sch
  • forwardest mirror position.

• 180205 03:02:21 - 180205 10:49:44

  • /home/keck/sch/mce_autotune.sch
  • /data/keckdaq/autoqueue/pending/7_ffflat_dslmast_33i_010.sch
  • forwardest mirror position.

• 180204 03:57:01 - 180204 11:39:02

  • /home/keck/sch/mce_autotune.sch
  • /data/keckdaq/autoqueue/pending/7_ffflat_dslmast_33h_009.sch
  • forwardest mirror position.

• 180203 14:48:01 - 180203 22:27:46

  • /home/keck/sch/mce_autotune.sch
  • /data/keckdaq/autoqueue/pending/7_ffflat_dslmast_33g_009.sch
  • forwardest mirror position.

• 180202 01:38:18 - 180202 9:18:21

  • /home/keck/sch/mce_autotune.sch
  • /data/keckdaq/autoqueue/pending/7_ffflat_dslmast_33f_009.sch
  • forwardest mirror position.

• 180202 09:14:33 - 180202 16:07:58

  • /home/keck/sch/mce_autotune.sch
  • /data/keckdaq/autoqueue/pending/7_ffflat_dslmast_33e_009.sch
  • forwardest mirror position.
  • Forgot to change the schedule name to 010.
  • Mediator crash near the end of the schedule.

• 180201 19:41:16 - 180202 03:22:44

  • /home/keck/sch/mce_autotune.sch
  • /data/keckdaq/autoqueue/pending/7_ffflat_dslmast_33d_009.sch
  • forwardest mirror position.

• 180201 06:55:25 - 180201 14:40:44

  • /home/keck/sch/mce_autotune.sch
  • /data/keckdaq/autoqueue/pending/7_ffflat_dslmast_33c_009.sch
  • forwardest mirror position.
  • rx1 fridge ran out 1 hour into the schedule.
  • gcp crashed during last fridge cycle, and we only did a single cycle afterwards on rx1.

• 180130 19:15:55 - 180131 02:56:27

  • /home/keck/sch/mce_autotune.sch
  • /data/keckdaq/autoqueue/pending/7_ffflat_dslmast_33b_009.sch
  • forwardest mirror position.

• 180130 02:18:23 - 180130 09:58:39

  • /home/keck/sch/mce_autotune.sch
  • /data/keckdaq/autoqueue/pending/7_ffflat_dslmast_33a_009.sch
  • First schedule with forwardest mirror position.
  • First half of data might be affected by ARIANNA wifi signal.

• 180129 10:01:21 - 180129 17:41:51

  • /home/keck/sch/mce_autotune.sch
  • /data/keckdaq/autoqueue/pending/7_ffflat_dslmast_33j_010.sch
  • change the days of data kept on Keck32 from 10days to 3days. (/home/keckdata/bin/purge_data.pl)
  • Redo schedule j. All 10 schedules done. Time to move mirror from back position to forward or forwardest.

• 180128 20:35:58 - 180129 04:15:44

  • /home/keck/sch/mce_autotune.sch
  • /data/keckdaq/autoqueue/pending/7_ffflat_dslmast_33i_010.sch
  • Deleted some data from Keck32:/data/keckdaq/arc/.
  • Redo schedule i.

• 180128 00:01:43 - 180128 07:42:12

  • /home/keck/sch/mce_autotune.sch
  • /data/keckdaq/autoqueue/pending/7_ffflat_dslmast_33j_009.sch
  • run out of disk space. Have to redo this schedule and maybe the previous one.

• 180126 22:57:27 - 180127

  • /home/keck/sch/mce_autotune.sch
  • /data/keckdaq/autoqueue/pending/7_ffflat_dslmast_33i_009.sch
  • Keck32 run out of /data disk space? Error in log file.

• 180126 02:18:45 - 180126 10:02:24

  • /home/keck/sch/mce_autotune.sch
  • /data/keckdaq/autoqueue/pending/7_ffflat_dslmast_33h_008.sch

• 180125 04:37:45 - 180125 12:17:52

  • /home/keck/sch/mce_autotune.sch
  • /data/keckdaq/autoqueue/pending/7_ffflat_dslmast_33g_008.sch

• 180124 10:30:11 - 180124 18:10:12

  • /home/keck/sch/mce_autotune.sch
  • /data/keckdaq/autoqueue/pending/7_ffflat_dslmast_33f_008.sch

• 180123 20:48:02 - 180124 04:28:58

  • /home/keck/sch/mce_autotune.sch
  • /data/keckdaq/autoqueue/pending/7_ffflat_dslmast_33e_009.sch

• 180123 07:34:49 - 180123 15:16:38

  • /home/keck/sch/mce_autotune.sch
  • /data/keckdaq/autoqueue/pending/7_ffflat_dslmast_33d_008.sch

• 180122 04:45:12 - 180122 12:25:48

  • /home/keck/sch/mce_autotune.sch
  • /data/keckdaq/autoqueue/pending/7_ffflat_dslmast_33c_008.sch

• 180121 06:36:14 - 180121 14:16:09

  • /home/keck/sch/mce_autotune.sch
  • /data/keckdaq/autoqueue/pending/7_ffflat_dslmast_33b_008.sch

• 180120 05:58:29 - 180120 13:38:27

  • /data/keckdaq/autoqueue/pending/7_ffflat_dslmast_33a_008.sch
  • First ffbm schedule with broadband noise source.
  • Fridge might have run out early for some rxs. (rx0, rx3)

• 180114 06:40:23 - 180114 06:51:19

  • /data/keckdaq/autoqueue/pending/1_mirror_noise_1_023.sch
  • All rx staring at blanking plate at nominal Ti bias, taking 2min each of combine off data five times.
  • K5 HK backpack now has capacitors
  • Raw data: rx4_hk_014
  • Discard 1_mirror_noise_1_024.sch

• 180110 00:56:08 - 180110 01:07:04

  • /data/keckdaq/autoqueue/pending/1_mirror_noise_1_021.sch
  • All rx staring at blanking plate at nominal Ti bias, taking 2min each of combine off data five times.
  • K5 HK backpack out for a couple hours now

• 180109 01:56:34 - 180109 02:07:30

  • /data/keckdaq/autoqueue/pending/1_mirror_noise_1_019.sch
  • All rx staring at blanking plate at nominal Ti bias, taking 2min each of combine off data five times.
  • All HK cables plugged back in.
  • Taken raw data at 50 MHz through MCE. keck37:/home/keck/raw/data_1_mirror_noise_1_019
  • Taken data through $ mce_run 1_mirror_noise_1_019_001 21645 s

• 180108 22:46:48 - 180108 22:57:45

  • /data/keckdaq/autoqueue/pending/1_mirror_noise_1_018.sch
  • All rx staring at blanking plate at nominal Ti bias, taking 2min each of combine off data five times.
  • All HK (for all rxs) power and digital cables disconnected.
  • Taken raw data at 50 MHz through MCE. keck37:/home/keck/raw/data_1_mirror_noise_1_018

• 180108 22:24:56 - 180108 22:35:52

  • /data/keckdaq/autoqueue/pending/1_mirror_noise_1_017.sch
  • All rx staring at blanking plate at nominal Ti bias, taking 2min each of combine off data five times.
  • All HK connected
  • Taken raw data at 50 MHz through MCE. keck37:/home/keck/raw/data_1_mirror_noise_1_017

• 180108 08:17:47 - 180108 08:28:43

  • /data/keckdaq/autoqueue/pending/1_mirror_noise_1_016.sch
  • All rx staring at blanking plate at nominal Ti bias, taking 2min each of combine off data five times.
  • K5 HK power cable, digital cable disconnected. MDM-100 disconnected.
  • Taken raw data at 50 MHz through MCE. keck37:/home/keck/raw/data_1_mirror_noise_1_016

• 180108 07:59:35 - 180108 08:10:32

  • /data/keckdaq/autoqueue/pending/1_mirror_noise_1_015.sch
  • Ignore 1_mirror_noise_1_011.sch - Ignore 1_mirror_noise_1_014.sch.
  • All rx staring at blanking plate at nominal Ti bias, taking 2min each of combine off data five times.
  • K5 HK fully connected.
  • Taken raw data at 50 MHz through MCE. keck37:/home/keck/raw/data_1_mirror_noise_1_015

• 180106 04:04:07 - 180106 04:04:07

  • /data/keckdaq/autoqueue/pending/1_mirror_noise_1_010.sch
  • All rx staring at blanking plate at nominal Ti bias, taking 2min each of combine off data five times.
  • Removed all K5 HK cables (power + A19 + A20 + digital cable)

• 180106 03:53:11 - 180106 04:04:07

  • /data/keckdaq/autoqueue/pending/1_mirror_noise_1_009.sch
  • All rx staring at blanking plate at nominal Ti bias, taking 2min each of combine off data five times.
  • All HK on. Placed ferrite bead on K5 HK digital cable.
  • Did not start the first time copied to /data/keckdaq/autoqueue/pending/

• 180106 03:32:36 - 180106 03:43:32

  • /data/keckdaq/autoqueue/pending/1_mirror_noise_1_008.sch
  • All rx staring at blanking plate at nominal Ti bias, taking 2min each of combine off data five times.
  • All HK connected back. Waited for temperature to settle. This is more of a consistency check measurement.

• 180106 02:20:58 - 180106 02:31:54

  • /data/keckdaq/autoqueue/pending/1_mirror_noise_1_007.sch
  • All rx staring at blanking plate at nominal Ti bias, taking 2min each of combine off data five times.
  • K5 HK, A19 and A20 cables are disconnected. Power and digital cable still on.

• 180106 01:45:14 - 180106 01:56:10

  • /data/keckdaq/autoqueue/pending/1_mirror_noise_1_006.sch
  • All rx staring at blanking plate at nominal Ti bias, taking 2min each of combine off data five times.
  • HK is on.
  • rx4 experiment.cfg: back to original, row_len = 60; sample_dly = 50; sample_num = 10;

• 180105 10:58:37 - 180105 11:09:33

  • /data/keckdaq/autoqueue/pending/1_mirror_noise_1_005.sch
  • All rx staring at blanking plate at nominal Ti bias, taking 2min each of combine off data five times.
  • HK is on.
  • rx4 experiment.cfg: row_len = 60; sample_dly = 54; sample_num = 6;

• 180105 10:12:54 - 180105 10:23:50

  • /data/keckdaq/autoqueue/pending/1_mirror_noise_1_004.sch
  • All rx staring at blanking plate at nominal Ti bias, taking 2min each of combine off data five times.
  • HK is on.
  • rx4 experiment.cfg: row_len = 60; sample_dly = 48; sample_num = 12;
  • Did not start the first time copied to /data/keckdaq/autoqueue/pending/

• 180105 02:41:27 - 180105 02:52:23

  • /data/keckdaq/autoqueue/pending/1_mirror_noise_1_003.sch
  • All rx staring at blanking plate at nominal Ti bias, taking 2min each of combine off data five times.
  • HK is on.
  • rx4 experiment.cfg: row_len = 60; sample_dly = 50; sample_num = 10;

• 180103 10:35:48 - 180103 20:16:05

  • 9_kCMB_50_258.sch
  • another test CMB schedule, to see if HK-induced 4 MHz we recently detected is
  • ...the source of row-correlated noise.
  • first scanset with normal HK operation.
  • after trailing elnod and before second scanset, unplugged HK backpack power and HK digital cables UTC 12:15-12:18

• 180102 16:19:25 - 180103 00:12:35

  • 9_kCMB_50_257.sch
  • test CMB schedule -- Justin and Marion to fill in details!!

• 180102 10:15:04 - 180102 16:19:25

  • 1_knoise_responsivity_9_023.sch
  • NET vs bias schedule, beautiful weather

• 180102 05:39:26 - 180102 07:33:03

  • 1_forebaffle_on_off_material_1_002.sch
  • Forebaffle on/off sequence on rx4 (270 GHz)
  • Similar sequence and same set of samples as _001.sch schedule below

• 171231 05:16:11 - 180101 19:34:17

  • 9_kCMB_50_256.sch
  • test CMB schedule after reinstalling rx4 (270 GHz)
  • Initial scansets used incorrect rx4 SQ tuning (experiment.cfg not updated)
  • rx4 also had elevated FPU temp due to PID control problems, tracked to bad heater 3
  • mid-schedule, switched PIT for rx4 to clean-side only which solved that problem
  • but rx4 TES biases were too low (had been set on mirror, not sky), so most latched

• 171222 07:29:03 - 171222 07:39:59

  • /data/keckdaq/autoqueue/pending/1_mirror_noise_1_002.sch
  • Rx0,2,3 staring at zenith at nominal Ti bias, taking 2min each of combine off data five times. FPUs a little hot at start of schedule, near ~310mK.

• 171221 22:45:51 - 171221 22:56:47

  • /data/keckdaq/autoqueue/pending/1_mirror_noise_1_001.sch
  • Rx0,2,3 with blanking plate at nominal Ti bias, taking 2min each of combine off data five times.

• 171219 02:41:08 - 171219 05:20:28

  • /home/keck/sch/1_forebaffle_on_off_material_1_001.sch
  • Repeat of measurements of materials' scattering using forebaffle on/off comparisons, but based on the 1_forebaffle_on_off_4_010.sch found in the autoqueue/done directory.
  • Sequence is generally:
    1. Sequence of 5 sec records with forebaffle off, on, off, on, off. Only using tallest forebaffle.
    2. Partial loadcurve immediately following with forebaffle off, then PLC with forebaffle on.
    3. Extra PLC we chose to ignore (supposed to be with forebaffle off, but we used it to change out materials).
  • In order, measured:
    1. Two leading elnods
    2. Zotefoam puck (bonded, from Caltech)
    3. Reference for Zotefoam measurements (no material)
    4. Zotefoam stack (4 layers, unbonded, from Caltech)
    5. Zotefoam puck (bonded, spare at Pole)
    6. 8.5 icm edge filter
    7. Reference for edge filter measurements (no material)
    8. 10.5 icm edge filter
    9. 5.5 icm edge filter
    10. Double layer Dyneema membrane
    11. Reference for Dyneema measurements (no material)
    12. Raw Dyneema weave membrane
    13. Single layer Dyneema membrane
  • Schedule was written to have more references, so aborted schedule to manually perform two trailing elnods. Search for 1_forebaffle_on_off_material_1_001.sch end time will not contain trailing elnod, but time given at the top of this entry does include that extra mini-schedule.

• 171208 22:52:15 - 171208 23:24:27

  • /home/keck/sch/forebaffle_on_off_material_scattering.sch
  • Detectors on Ti transition
  • Each material measurement a sequence of 4 measurements.
  • Sequence of material-only, material+forebaffle, nothing, forebaffle-only measurements on rx2 — corresponds to feature bits 0, 1, 2, and 3, respectively.
  • Materials in order:
    1. Zotefoam window (Keck style), bonded (4 layers of laminates)
    2. Zotefoam window (Keck style), unbonded
    3. 8.5 icm edge filter ( + Eccosorb surround)
    4. 10.5 icm edge filter ( + Eccosorb surround)
  • Same non-idealities as previous schedule (minus wooden frame since no Dyneema materials tested)

• 171208 21:39:51 - 171208 22:29:41

  • /home/keck/sch/forebaffle_on_off_material_scattering.sch
  • Detectors on Ti transition
  • Each material measurement a sequence of 4 measurements.
  • Sequence of material-only, material+forebaffle, nothing, forebaffle-only measurements on rx2 — corresponds to feature bits 0, 1, 2, and 3, respectively.
  • Materials in order:
    1. Single layer laminated Dyneema
    2. Zotefoam window (Keck style), bonded (4 layers of laminates)
    3. Raw Dyneema (single layer)
    4. Two layer laminated Dyneema
  • Non-idealities of note:
    • Open ground shield
    • People on top of mount
    • Ladder top visible above top of mount
    • Wooden stretcher frame has to pass over cryostat window / through beams

• 171119 21:34:58

  • Turning off rx4 compressor, putting on turbo, to start warmup
  • 171119 22:10:53 Turning on rx4 fridge pumps to force thermal bridge (based on cooldown instructions, but interpreted in reverse)
  • 171119 22:17:53 Starting fridge cycle on other rxs
  • 171121 ~21:00 recharged compressors in stations 2 and 4

• • cyclefridge enable=2, on all rx
  • recharged compressor 2
  • replaced N2 bottle for membranes

• 171115 04:45:18 -

  • Using this schedule to test out various GCP changes and mediator behaviors, so expect timing and data consistency to be poor.
  • Started late as restarting GCP with new checkout to test cleaned-up-for-committability of Grantland's patches.
  • /home/keck/sch/mce_autotune.sch
  • check_rms.py on mce4 = 3.351
  • /data/keckdaq/autoqueue/pending/9_kCMB_50_255.sch
  • /home/keck/sch/runend.sch
  • weather: clear
  • windows and baffles:

• 171114 22:13:28 - 171115 04:43:19

  • cyclefridge enable=2, on all rx

• 171113 03:51:31 - 171114 22:13:28

  • /home/keck/sch/mce_autotune.sch
  • check_rms.py on mce4 = 2.82
  • /data/keckdaq/autoqueue/pending/9_kCMB_50_254.sch
  • El currents diverged, but was away/sleeping. Eventually pmac program stopped running, but neither caused pager alarms to sound.
  • Added strong (bolded) statement to "Keck CMB and Run End Procedures" cookbook about watching motor currents at start of each run.
  • 171113 20:54:55 El motor currents started diverging again. Stopped pmac program, opened servo loops, closed loops, and restarted pmac program.
  • 171113 23:10:16 Starting memory write stressors
    Running with 4GB consumed memory (out of 8GB)
    top reports ~only 50MB free, but still not swapping
  • 171113 23:20:35 ended 4GB test (runtime ~10 minutes)
  • 171113 23:24:19 Running at 7GB (out of 8GB)
  • Mediator crashed!
  • Try no-write, locked-mmap buffers to force memory pressure without CPU cycles.
  • 171114 01:44:00 Starting locked memory test @ 7GB
  • mem_locked OOM killed; mediator and control killed/crashed too
  • 171114 03:02:00 Discover that no external programs needed to induce a mediator stall. To clearly see the entire mediator stall, try:
    
    1. Start the mediator
    2. Get mediator PID with ps -ef | grep [k]eckMediator.
    3. Attach to it with strace: strace -f -p <PID> -e trace='!write'
    4. Note that multiple PID/thread ids are listed in the output
    5. Now click and hold with a mouse button on the mediator xterm window; the text will stop scrolling, but if held long enough, the strace output will also stop on all threads.
    6. Release the mouse button. Wait a short bit (order 10 seconds) to see if mediator has stalled; you should see a long series of futex syscalls in a row. If not, click and hold longer and repeat.
    7. Eventually mediator will stall with several "Got an output frame!" messages showing up last in the window, which Robert has reported seeing in the past when the mediator stops.
  • weather: clear, mostly clear
  • windows and baffles:

• 171112 22:21:20 - 171113 03:51:31

  • cyclefridge enable=2, on all rx

• 171111 04:37:19 - 171112 22:18:09

  • /home/keck/sch/mce_autotune.sch
  • check_rms.py on mce4 = 2.651
  • /data/keckdaq/autoqueue/pending/9_kCMB_50_253.sch
  • keck32 has 4 cores; going to do CPU-only stress tests, incrementally increasing the number of cores which are used doing computations
  • 171112 07:07:34 Starting CPU stress test on keck32 (single thread) — ran for ~7 minutes
  • 171112 07:23:44 Starting CPU stress test on keck32 with 2 threads — ran for ~10 minutes
  • 171112 07:33:52 Starting CPU stress test on keck32 with 3 threads — ran for ~10 minutes
  • 171112 07:43:20 Starting CPU stress test on keck32 with 4 threads — ran for ~10 minutes
  • 4 threads test ocurring during scanset E04
  • 171112 07:53:15 End of CPU stress tests on keck32
  • /home/keck/sch/runend.sch
  • weather: blowing snow, clear
  • windows and baffles: clear

• 171110 22:29:11 - 171111 04:37:19

  • cyclefridge enable=2, on all rx
  • refill compressor station 2 (rx1) from HI 283PSI to 287PSI

• 171109 04:05:54 - 171110 22:26:01

  • /home/keck/sch/mce_autotune.sch
  • /data/keckdaq/autoqueue/pending/9_kCMB_50_252.sch
  • 171110 02:42:59 echo "Pager was activated (by register antenna0.pmac.drive_status[0])"
    No faults on drives - pmac program showed as stopped in pmac.page
    Doing `b1r` at pmacterm restored motion
  • 171110 02:49:14 started rebuild of keck35 RAID
  • /home/keck/sch/runend.sch
  • weather: blowing snow, lots of blowing snow, still blowing snow
  • Snow concentrated in ground shield only on up-wind side, but enough that environmental boot started to sweep out gouge into drift
  • windows and baffles: windows clear, couple handfulls of snow in each baffle

• 171108 22:37:10 - 171109 04:05:54

  • cyclefridge enable=2, on all rx
  • 171109 00:48:53 restarted masd2 since clock counter didn't stop with runend.sch
  • 171109 04:27:33 check_rms.py on rx4 returns a value of 2.694

• 171107 04:37:17 - 171108 22:34:00

  • /home/keck/sch/mce_autotune.sch
  • /data/keckdaq/autoqueue/pending/9_kCMB_50_251.sch
  • /home/keck/sch/runend.sch
  • weather: thin clouds, clear, blowing snow
  • windows and baffles: light snow in and on baffles

• 171106 22:44:54

  • cyclefridge enable=2, on all rx

• 171105 04:45:09 - 171106 22:44:54

  • /home/keck/sch/mce_autotune.sch
  • /data/keckdaq/autoqueue/pending/9_kCMB_50_250.sch
  • /home/keck/sch/runend.sch
  • weather: clear overhead, blowing snow, overcast
  • windows and baffles: windows clear, some snow in the baffles

• 171104 22:57:32

  • /home/keck/sch/mce_autotune.sch
  • check_rms.py 3.404
  • removed HK power on rx4, removed connectors, reseated connectors, turned power back on
  • /home/keck/sch/mce_autotune.sch
  • check_rms.py 2.688
  • cyclefridge enable=2, on all rx
  • recharged comp 2,3

• 171103 04:32:35 - 171104 22:52:49

  • /home/keck/sch/mce_autotune.sch
  • /data/keckdaq/autoqueue/pending/9_kCMB_50_249.sch
  • pmac stopped
  • mediator stopped
  • 171104 10:38:31 Starting schedule: /home/keck/sch/mce_autotune.sch
  • 171104 10:40:21 Starting schedule: /data/keckdaq/autoqueue/pending/9_kCMB_50_249.sch
  • /home/keck/sch/runend.sch
  • weather: clear, clear overhead, blowing snow
  • windows and baffles: clear

• 171102 23:00:40

  • cyclefridge enable=2, on all rx
  • recharged compressor 4

• 171101 05:00:52 - 171102 23:00:40

  • /home/keck/sch/mce_autotune.sch
  • /data/keckdaq/autoqueue/pending/9_kCMB_50_248.sch
  • pmac stopped
  • mediator stopped
  • restarted gcp and masd
  • 171101 10:10:23 Starting schedule: /home/keck/sch/mce_autotune.sch
  • 171101 10:12:12 Starting schedule: /data/keckdaq/autoqueue/pending/9_kCMB_50_248.sch
  • /home/keck/sch/runend.sch
  • weather: clear
  • windows and baffles: clear

• 171031 23:08:29

  • cyclefridge enable=2, on all rx
  • swapped mce4 and mce1 back to original positions

• 171030 05:20:50 - 171031 23:08:29

  • /home/keck/sch/mce_autotune.sch
  • /data/keckdaq/autoqueue/pending/9_kCMB_50_247.sch
  • mediator stopped
  • restarted gcp and masd
  • 171030 19:43:11 Starting schedule: /home/keck/sch/mce_autotune.sch
  • 171030 19:45:01 Starting schedule: /data/keckdaq/autoqueue/pending/9_kCMB_50_247.sch
  • /home/keck/sch/runend.sch
  • weather: clear, clear overhead, clear
  • windows and baffles: clear

• 171030 00:03:16

  • cyclefridge enable=2, on all rx
  • 171030 02:30:08 Starting schedule: /home/keck/sch/starpoint_pole_keck_summer.sch
  • 171030 04:32:30 Exiting schedule: /home/keck/sch/starpoint_pole_keck_summer.sch

• 171028 05:16:26 - 171030 00:03:16

  • /home/keck/sch/mce_autotune.sch
  • /data/keckdaq/autoqueue/pending/9_kCMB_50_246.sch
  • 171029 01:47:40 masd2 stopped
  • restarted keck35, masd2
  • mediator stopped
  • restarted gcp and masd
  • 171029 06:42:16 Starting schedule: /home/keck/sch/mce_autotune.sch
  • 171029 06:46:25 Starting schedule: /data/keckdaq/autoqueue/pending/9_kCMB_50_246.sch
  • masd2 not working again
  • had to restart keck35 again
  • mediator stopped
  • restarted gcp and masd
  • 171029 10:25:53 Starting schedule: /home/keck/sch/mce_autotune.sch
  • masd2 not working, disk problems on keck35
  • mediator stopped
  • restarted gcp and masd
  • weather: mostly clear, thin clouds, overcast, clear
  • windows and baffles: clear

• 171027 23:33:47

  • cyclefridge enable=2, on all rx
  • overcast, no starpointing

• 171026 05:26:44 - 171027 23:33:47

  • /home/keck/sch/mce_autotune.sch
  • /data/keckdaq/autoqueue/pending/9_kCMB_50_245.sch
  • mediator stopped
  • restarted gcp and masd
  • 171026 08:57:08 Starting schedule: /home/keck/sch/mce_autotune.sch
  • 171026 08:58:57 Starting schedule: /data/keckdaq/autoqueue/pending/9_kCMB_50_245.sch
  • mediator stopped, at least I assume it was mediator, because without the windows it's hard to tell
  • restarted gcp and masd
  • 171026 22:13:24 Starting schedule: /home/keck/sch/mce_autotune.sch
  • 171026 22:15:14 Starting schedule: /data/keckdaq/autoqueue/pending/9_kCMB_50_245.sch
  • gcp stopped AGAIN
  • restarted gcp and masd
  • 171027 13:04:50 Starting schedule: /home/keck/sch/mce_autotune.sch
  • 171027 13:06:40 Starting schedule: /data/keckdaq/autoqueue/pending/9_kCMB_50_245.sch
  • 171027 23:30:36 abort_schedule, didn't change schedule
  • /home/keck/sch/runend.sch
  • weather: overcast, warm
  • windows and baffles: windows mostly clear, ice build up around the venting holes on top of membrane, some snow in and on the baffles
  • hole in membrane A, rx0, while trying to remove ice

• 171026 00:02:06

  • cyclefridge enable=2, on all rx
  • overcast, no starpointing
  • rx4 also got 2 pins
  • restarted gcp and masd

• 171024 05:32:19 - 171026 00:02:06

  • /home/keck/sch/mce_autotune.sch
  • /data/keckdaq/autoqueue/pending/9_kCMB_50_244.sch
  • pmac stopped
  • mediator stopped
  • restarted gcp and masd
  • 171025 06:33:59 Starting schedule: /home/keck/sch/mce_autotune.sch
  • 171025 06:35:53 Starting schedule: /data/keckdaq/autoqueue/pending/9_kCMB_50_244.sch
  • forgot to change schedule, so it started with phase B again
  • 171025 23:58:53 abort_schedule
  • /home/keck/sch/runend.sch
  • weather: thin clouds, overcast
  • windows and baffles: windows clear, lots of snow in and on baffles

• 171023 23:39:59

  • cyclefridge enable=2, on all rx
  • pmac stopped
  • recharged comp 2, not as high as before, does that make any difference?
  • thin clouds, no starpointing

• 171022 05:46:33 - 171023 23:39:59

  • /home/keck/sch/mce_autotune.sch
  • /data/keckdaq/autoqueue/pending/9_kCMB_50_243.sch
  • az-diff jump
  • 171022 12:51:13 suspend_schedule
  • 171022 13:03:31 resume_schedule
  • 171023 01:12:39 abort_schedule
  • worked again on the crate, it was physically impossible to get them mated, removed one pin
  • from backplate and press fitted it into the crate
  • 171023 01:59:07 Starting schedule: /home/keck/sch/mce_autotune.sch
  • 171023 02:02:38 Starting schedule: /data/keckdaq/autoqueue/pending/9_kCMB_50_243.sch
  • rx1 working again
  • /home/keck/sch/runend.sch
  • weather: overcast, blowing snow, thin clouds, wind dying off
  • windows and baffles: windows clear, very little snow in and on the baffles

• 171021 23:47:51

  • cyclefridge enable=2, on all rx
  • pmac stopped

• 171020 05:26:14 - 171021 23:47:51

  • /home/keck/sch/mce_autotune.sch
  • /data/keckdaq/autoqueue/pending/9_kCMB_50_242.sch
  • mediator stopped
  • restarted gcp and masd
  • 171020 09:11:44 Starting schedule: /home/keck/sch/mce_autotune.sch
  • 171020 09:13:34 Starting schedule: /data/keckdaq/autoqueue/pending/9_kCMB_50_242.sch
  • /home/keck/sch/runend.sch
  • weather: thin clouds, blowing snow, high winds, overcast
  • windows and baffles: clear

• 171019 23:55:43

  • cyclefridge enable=2, on all rx
  • thin clouds, no starpointing

• 171018 06:22:24 - 171019 23:55:43

  • /home/keck/sch/mce_autotune.sch
  • /data/keckdaq/autoqueue/pending/9_kCMB_50_241.sch
  • pmac stopped
  • rx1 doesn't show any read out on sample bolos, but masd1 writing frme numbers
  • 171019 00:56:51 abort_schedule
  • try to reseat MCE1 and connectors several times, but always the same result
  • 171019 01:56:24 Starting schedule: /home/keck/sch/mce_autotune.sch
  • 171019 01:58:12 Starting schedule: /data/keckdaq/autoqueue/pending/9_kCMB_50_241.sch
  • /home/keck/sch/runend.sch
  • weather: thin clouds, overcast, storm, blowing snow
  • windows and baffles: windows clear, some snow in and on the baffles

• 171018 00:03:34

  • cyclefridge enable=2, on all rx
  • swapped MCE rx1 with MCE rx4
  • thin clouds, no starpointing
  • restarted masd
  • pmac stopped

• 171016 05:37:27 - 171018 00:03:34

  • /home/keck/sch/mce_autotune.sch
  • /data/keckdaq/autoqueue/pending/9_kCMB_50_240.sch
  • /home/keck/sch/runend.sch
  • weather: thin clouds
  • windows and baffles: windows clear, little snow on the baffles

• 171016 00:11:27

  • cyclefridge enable=2, on all rx

• 171014 06:11:39 - 171016 00:11:27

  • /home/keck/sch/mce_autotune.sch
  • /data/keckdaq/autoqueue/pending/9_kCMB_50_239.sch
  • /home/keck/sch/runend.sch
  • weather: thin clouds
  • windows and baffles: windows clear, little snow on the baffles

• 171014 00:19:17

  • cyclefridge enable=2, on all rx

• 171012 06:16:13 - 171014 00:19:17

  • /home/keck/sch/mce_autotune.sch
  • /data/keckdaq/autoqueue/pending/9_kCMB_50_238.sch
  • /home/keck/sch/runend.sch
  • weather: thin clouds
  • windows and baffles: clear

• 171012 00:27:10

  • keck32 unresponsive restarted keck32, gcp, masd
  • cyclefridge enable=2, on all rx

• 171010 06:27:22 - 171012 00:27:10

  • /home/keck/sch/mce_autotune.sch
  • /data/keckdaq/autoqueue/pending/9_kCMB_50_237.sch
  • /home/keck/sch/runend.sch
  • weather: thin clouds
  • windows and baffles:

• 171010 01:13:58

  • cyclefridge enable=2, on all rx
  • recharged comp 4

• 171008 06:35:14 - 171010 00:35:00

  • /home/keck/sch/mce_autotune.sch
  • /data/keckdaq/autoqueue/pending/9_kCMB_50_236.sch
  • /home/keck/sch/runend.sch
  • weather: mostly clear, thin clouds
  • windows and baffles: clear

• 171008 00:42:52

  • cyclefridge enable=2, on all rx

• 171006 06:43:06 - 171008 00:42:52

  • /home/keck/sch/mce_autotune.sch
  • /data/keckdaq/autoqueue/pending/9_kCMB_50_235.sch
  • /home/keck/sch/runend.sch
  • weather: mostly clear
  • windows and baffles: clear

• 171006 00:50:45

  • cyclefridge enable=2, on all rx
  • restarted masd

• 171004 06:23:01 - 171006 00:50:45

  • /home/keck/sch/mce_autotune.sch
  • /data/keckdaq/autoqueue/pending/9_kCMB_50_234.sch
  • /home/keck/sch/runend.sch
  • weather: haze, wind picking up, storm, blowing snow, clearing, mostly clear
  • windows and baffles: windows clear, some snow in the baffles

• 171004 01:00:24

  • cyclefridge enable=2, on all rx
  • recharged comp 3

• 171002 06:43:44 - 171004 01:00:24

  • /home/keck/sch/mce_autotune.sch
  • /data/keckdaq/autoqueue/pending/9_kCMB_50_233.sch
  • /home/keck/sch/runend.sch
  • weather: haze overhead, clearing, clear
  • windows and baffles: clear

• 171002 01:06:28

  • cyclefridge enable=2, on all rx
  • recharged comp 2

• 170930 06:54:45 - 171002 01:06:28

  • /home/keck/sch/mce_autotune.sch
  • /data/keckdaq/autoqueue/pending/9_kCMB_50_232.sch
  • /home/keck/sch/runend.sch
  • weather: thin clouds, haze
  • windows and baffles: windows clear, little snow in and on the baffles

• 170930 01:14:19

  • cyclefridge enable=2, on all rx

• 170928 06:44:48 - 170930 01:14:19

  • /home/keck/sch/mce_autotune.sch
  • /data/keckdaq/autoqueue/pending/9_kCMB_50_231.sch
  • 170928 15:30:11masd2 stopped, restarted masd2
  • 170929 15:26:15 masd0 stopped, restarted masd0
  • clk count was off for mce0 and mce2
  • 170929 16:54:36 restarted all masd
  • clk count back in sync
  • /home/keck/sch/runend.sch
  • weather: overcast, blowing snow, high winds, wind easing, thin clouds
  • windows and baffles: windows clear, lots of snow in and on the baffles

• 170928 01:22:12

  • cyclefridge enable=2, on all rx

• 170926 07:22:25 - 170928 01:22:12

  • /home/keck/sch/mce_autotune.sch
  • /data/keckdaq/autoqueue/pending/9_kCMB_50_230.sch
  • mediator stopped
  • restarted gcp and masd
  • 170927 04:38:45 Starting schedule: /home/keck/sch/mce_autotune.sch
  • 170927 04:41:44 Starting schedule: /data/keckdaq/autoqueue/pending/9_kCMB_50_230.sch
  • /home/keck/sch/runend.sch
  • weather: overcast, blowing snow, high winds
  • windows and baffles: windows clear, some snow on a ton in the baffles, most I ever seen

• 170926 01:32:16

  • cyclefridge enable=2, on all rx

• 170924 07:30:17 - 170926 01:30:04

  • /home/keck/sch/mce_autotune.sch
  • /data/keckdaq/autoqueue/pending/9_kCMB_50_229.sch
  • /home/keck/sch/runend.sch
  • data transfer caught up
  • restarted gcp and masd
  • weather: overcast, blowing snow
  • windows and baffles: windows clear, some snow on the baffles

• 170924 01:37:56

  • cyclefridge enable=2, on all rx
  • problems with omega0 system disk, replaced for now with last years disk
  • data handling 30h behind, will take a bit to catch up
  • signalhound and WVR data transfer compromised for now

• 170922 07:14:20 - 170924 01:37:56

  • /home/keck/sch/mce_autotune.sch
  • /data/keckdaq/autoqueue/pending/9_kCMB_50_228.sch
  • /home/keck/sch/runend.sch
  • weather: clear, overcast, blowing snow
  • windows and baffles: windows clear, some snow in and on the baffles

• 170922 01:50:32

  • cyclefridge enable=2, on all rx
  • el current drifting apart

• 170920 07:21:09 - 170922 01:45:46

  • /home/keck/sch/mce_autotune.sch
  • /data/keckdaq/autoqueue/pending/9_kCMB_50_227.sch
  • mediator stopped
  • restarted gcp and masd
  • 170921 02:42:17 Starting schedule: /home/keck/sch/mce_autotune.sch
  • 170921 02:44:07 Starting schedule: /data/keckdaq/autoqueue/pending/9_kCMB_50_227.sch
  • /home/keck/sch/runend.sch
  • weather: mostly clear, some thin clouds
  • windows and baffles: clear

• 170920 01:53:38

  • cyclefridge enable=2, on all rx
  • 170920 04:46:05 Starting schedule: /home/keck/sch/starpoint_pole_keck_summer.sch
  • 170920 06:43:07 Exiting schedule: /home/keck/sch/starpoint_pole_keck_summer.sch
  • running gcp and masd without keckViewer windows just in terminals

• 170918 07:22:11 - 170920 01:53:38

  • /home/keck/sch/mce_autotune.sch
  • /data/keckdaq/autoqueue/pending/9_kCMB_50_226.sch
  • /home/keck/sch/runend.sch
  • weather: mostly clear overhead, thin clouds, clearing
  • windows and baffles: windows clear, some snow in and on the baffles

• 170918 02:01:31

  • cyclefridge enable=2, on all rx
  • recharged comp 1,3

• 170916 08:01:44 - 170918 02:01:31

  • /home/keck/sch/mce_autotune.sch
  • /data/keckdaq/autoqueue/pending/9_kCMB_50_225.sch
  • /home/keck/sch/runend.sch
  • weather: clear overhead, blowing snow
  • windows and baffles: clear

• 170916 02:09:23

  • cyclefridge enable=2, on all rx
  • recharged comp 4
  • running all keckViewers on keck31 none on keck32

• 170914 07:46:06 - 170916 02:09:23

  • /home/keck/sch/mce_autotune.sch
  • /data/keckdaq/autoqueue/pending/9_kCMB_50_224.sch
  • 170915 21:28 LMR attenuation is off
  • mediator stopped
  • restarted gcp and masd
  • 170916 00:16:44 Starting schedule: /home/keck/sch/mce_autotune.sch
  • 170916 00:18:33 Starting schedule: /data/keckdaq/autoqueue/pending/9_kCMB_50_224.sch
  • /home/keck/sch/runend.sch
  • weather: mostly clear overhead, overcast, blowing snow, clearing
  • windows and baffles: clear

• 170914 02:20:33

  • cyclefridge enable=2, on all rx

• 170912 08:17:28 - 170914 02:20:33

  • /home/keck/sch/mce_autotune.sch
  • /data/keckdaq/autoqueue/pending/9_kCMB_50_223.sch
  • mediator stopped
  • restarted gcp and masd
  • 170912 14:11:43 Starting schedule: /home/keck/sch/mce_autotune.sch
  • 170912 14:13:33 Starting schedule: /data/keckdaq/autoqueue/pending/9_kCMB_50_223.sch
  • /home/keck/sch/runend.sch
  • weather: mostly clear overhead
  • windows and baffles:

• 170912 02:25:06

  • cyclefridge enable=2, on all rx
  • replaced bad He line with new spare
  • running 2 keckViewers on keck31 instead of keck32

• 170910 08:25:19 - 170912 02:25:06

  • /home/keck/sch/mce_autotune.sch
  • /data/keckdaq/autoqueue/pending/9_kCMB_50_222.sch
  • /home/keck/sch/runend.sch
  • weather: clear, mostly clear overhead
  • windows and baffles: windows clear, very little snow on the baffles

• 170910 02:34:00

  • cyclefridge enable=2, on all rx
  • He leak B high, replaced with spare line

• 170908 08:43:10 - 170910 02:34:00

  • /home/keck/sch/mce_autotune.sch
  • /data/keckdaq/autoqueue/pending/9_kCMB_50_221.sch
  • mediator stopped
  • restarted gcp and masd
  • 170909 23:47:59 Starting schedule: /home/keck/sch/mce_autotune.sch
  • 170909 23:49:48 Starting schedule: /data/keckdaq/autoqueue/pending/9_kCMB_50_221.sch
  • /home/keck/sch/runend.sch
  • weather: overcast, high winds, blowing snow, clearing
  • windows and baffles: clear

• 170908 03:08:48

  • cold rebooted all keck computers
  • cyclefridge enable=2, on all rx
  • recharged comp 2
  • swapped air handler belt, installed belt ripped and compressors shut down due to high oil temperature
  • replaced belt and restarted compressors after a few minutes, doesn't look like any significant impact on
  • the temperatures

• 170906 08:25:56 - 170908 02:40:51

  • /home/keck/sch/mce_autotune.sch
  • /data/keckdaq/autoqueue/pending/9_kCMB_50_220.sch
  • mediator stopped
  • restarted gcp and masd
  • 170906 23:41:15 Starting schedule: /home/keck/sch/mce_autotune.sch
  • 170906 23:43:05 Starting schedule: /data/keckdaq/autoqueue/pending/9_kCMB_50_220.sch
  • keck37 froze
  • rebooted keck37 and restarted masd4
  • mediator stopped
  • restarted gcp and masd
  • 170907 23:38:23 Starting schedule: /home/keck/sch/mce_autotune.sch
  • 170907 23:40:13 Starting schedule: /data/keckdaq/autoqueue/pending/9_kCMB_50_220.sch
  • /home/keck/sch/runend.sch
  • weather: hazy, thin clouds, high winds, blowing snow
  • windows and baffles: windows clear, little snow in and on the baffles

• 170906 02:45:33

  • cyclefridge enable=2, on all rx
  • restarted gcp and masd

• 170904 08:48:55 - 170906 02:45:33

  • /home/keck/sch/mce_autotune.sch
  • /data/keckdaq/autoqueue/pending/9_kCMB_50_219.sch
  • mediator stopped
  • restarted gcp and masd
  • 170905 21:18:54 Starting schedule: /home/keck/sch/mce_autotune.sch
  • 170905 21:20:43 Starting schedule: /data/keckdaq/autoqueue/pending/9_kCMB_50_219.sch
  • /home/keck/sch/runend.sch
  • weather: mostly clear overhead, hazy
  • windows and baffles: windows clear, very little snow in the baffles

• 170904 02:56:35

  • cyclefridge enable=2, on all rx

• 170902 08:56:47 - 170904 02:56:35

  • /home/keck/sch/mce_autotune.sch
  • /data/keckdaq/autoqueue/pending/9_kCMB_50_218.sch
  • mediator stopped
  • restarted gcp and masd
  • 170903 00:13:31 Starting schedule: /home/keck/sch/mce_autotune.sch
  • 70903 00:17:18 Starting schedule: /data/keckdaq/autoqueue/pending/9_kCMB_50_218.sch
  • /home/keck/sch/runend.sch
  • weather: hazy
  • windows and baffles: windows clear, some snow in and on the baffles

• 170902 03:04:27

  • cyclefridge enable=2, on all rx
  • recharged comp 3

• 170831 08:45:31 - 170902 03:04:27

  • /home/keck/sch/mce_autotune.sch
  • /data/keckdaq/autoqueue/pending/9_kCMB_50_217.sch
  • mediator crashes are back :(((
  • restarted gcp and masd
  • 170902 00:20:26 Starting schedule: /home/keck/sch/mce_autotune.sch
  • 170902 00:22:20 Starting schedule: /data/keckdaq/autoqueue/pending/9_kCMB_50_217.sch
  • /home/keck/sch/runend.sch
  • weather: mostly clear overhead, hazy
  • windows and baffles: clear

• 170831 03:12:21

  • cyclefridge enable=2, on all rx

• 170829 08:46:02 - 170831 03:12:21

  • /home/keck/sch/mce_autotune.sch
  • /data/keckdaq/autoqueue/pending/9_kCMB_50_216.sch
  • /home/keck/sch/runend.sch
  • weather: mostly clear overhead, blowing snow
  • windows and baffles: windows clear, quite a bit of snow in and on the baffles

• 170829 03:20:09

  • cyclefridge enable=2, on all rx
  • recharged comp 4
  • restarted gcp, masd

• 170827 08:54:28 - 170829 03:20:09

  • /home/keck/sch/mce_autotune.sch
  • /data/keckdaq/autoqueue/pending/9_kCMB_50_215.sch
  • mediator stopped, restarted gcp and masd
  • 170828 10:48:58 Starting schedule: /data/keckdaq/autoqueue/pending/9_kCMB_50_215.sch
  • el current drifting apart couple of times
  • /home/keck/sch/runend.sch
  • weather: clear, cold, mostly clear overhead
  • windows and baffles: windows clear, some snow in and on the baffles

• 170827 03:28:01 Exiting

  • cyclefridge enable=2, on all rx
  • replaced readhead roller A2
  • 170827 05:21:32 Starting schedule: /home/keck/sch/starpoint_pole_keck_winter.sch
  • 170827 08:48:53 Exiting schedule: /home/keck/sch/starpoint_pole_keck_winter.sch

• 170825 09:28:14 - 170827 03:28:01

  • /home/keck/sch/mce_autotune.sch
  • /data/keckdaq/autoqueue/pending/9_kCMB_50_214.sch
  • az-diff jump, data are good until jump and after resume schedule
  • 170827 00:33:58 resume_schedule
  • /home/keck/sch/runend.sch
  • weather: mostly clear overhead, clear, cold
  • windows and baffles: clear

• 170825 03:35:53

  • cyclefridge enable=2, on all rx
  • recharged comp 2

• 170823 09:08:46 - 170825 03:35:53

  • /home/keck/sch/mce_autotune.sch
  • /data/keckdaq/autoqueue/pending/9_kCMB_50_213.sch
  • /home/keck/sch/runend.sch
  • weather: thin clouds
  • windows and baffles: windows clear, very little snow in the baffles

• 170823 03:43:44

  • cyclefridge enable=2, on all rx

• 170821 09:17:40 - 170823 03:43:44

  • /home/keck/sch/mce_autotune.sch
  • /data/keckdaq/autoqueue/pending/9_kCMB_50_212.sch
  • /home/keck/sch/runend.sch
  • weather: hazy, thin clouds, warm
  • windows and baffles: windows clear, some snow in and on the baffle

• 170821 03:51:37

  • cyclefridge enable=2, on all rx

• 170819 09:51:49 - 170821 03:51:37

  • /home/keck/sch/mce_autotune.sch
  • /data/keckdaq/autoqueue/pending/9_kCMB_50_211.sch
  • /home/keck/sch/runend.sch
  • weather: mostly clear overhead, very hazy
  • windows and baffles: windows clear, very little snow on the baffles

• 170819 04:00:31

  • cyclefridge enable=2, on all rx
  • recharged comp 3

• 170817 10:16:23 - 170819 04:00:31

  • /home/keck/sch/mce_autotune.sch
  • /data/keckdaq/autoqueue/pending/9_kCMB_50_210.sch
  • /home/keck/sch/runend.sch
  • weather: thin clouds, blowing snow, clearing, mostly clear overhead
  • windows and baffles: windows clear, some snow in the baffles

• 170817 04:07:22

  • cyclefridge enable=2, on all rx

• 170815 10:42:16 - 170817 04:07:22

  • /home/keck/sch/mce_autotune.sch
  • /data/keckdaq/autoqueue/pending/9_kCMB_50_209.sch
  • pmac stopped
  • 170816 01:45 changed E high hose, rx4 FPU temp warmed up sligh
  • /home/keck/sch/runend.sch
  • weather: hazy, thin clouds, high winds, blowing snow
  • windows and baffles: windows clear, tons of snow in the baffles

• 170815 04:15:14

  • cyclefridge enable=2, on all rx
  • possible leak on comp 5

• 170813 10:04:43 - 170815 04:15:14

  • /home/keck/sch/mce_autotune.sch
  • /data/keckdaq/autoqueue/pending/9_kCMB_50_208.sch
  • /home/keck/sch/runend.sch
  • weather: clear overhead, perfectly clear, cold, hazy
  • windows and baffles: windows clear, very litte snow on the baffles

• 170813 04:23:03

  • cyclefridge enable=2, on all rx

• 170811 10:27:10 - 170813 04:23:03

  • /home/keck/sch/mce_autotune.sch
  • /data/keckdaq/autoqueue/pending/9_kCMB_50_207.sch
  • /home/keck/sch/runend.sch
  • weather: clear overhead
  • windows and baffles: windows clear, very little snow on the baffles

• 170811 04:30:55

  • cyclefridge enable=2, on all rx
  • recharged comp 4

• 170809 10:00:21 - 170811 04:30:55

  • /home/keck/sch/mce_autotune.sch
  • /data/keckdaq/autoqueue/pending/9_kCMB_50_206.sch
  • /home/keck/sch/runend.sch
  • weather: hazy, clear overhead
  • windows and baffles: windows clear, a little bit of snow in and on the baffles

• 170809 04:38:49

  • cyclefridge enable=2, on all rx

• 170807 10:17:26 - 170809 04:38:49

  • /home/keck/sch/mce_autotune.sch
  • /data/keckdaq/autoqueue/pending/9_kCMB_50_205.sch
  • /home/keck/sch/runend.sch
  • weather: clear, hazy, blowing snow
  • windows and baffles: windows clear, lots of snow in and on the baffles

• 170807 04:46:39

  • cyclefridge enable=2, on all rx
  • 170807 06:37:22 Starting schedule: /home/keck/sch/starpoint_pole_keck_winter.sch
  • 170807 10:16:54 Exiting schedule: /home/keck/sch/starpoint_pole_keck_winter.sch

• 170805 10:22:49 - 170807 04:46:39

  • /home/keck/sch/mce_autotune.sch
  • /data/keckdaq/autoqueue/pending/1_test_lc_at_diff_fpu_temp.sch
  • 170805 12:33:15 Starting schedule: /data/keckdaq/autoqueue/pending/9_kCMB_50_204.sch
  • /home/keck/sch/runend.sch
  • weather: little hazy, clear
  • windows and baffles: clear

• 170805 04:54:30

  • cyclefridge enable=2, on all rx

• 170803 10:49:27 - 170805 04:54:30

  • /home/keck/sch/mce_autotune.sch
  • /data/keckdaq/autoqueue/pending/9_kCMB_50_203.sch
  • /home/keck/sch/runend.sch
  • weather: hazy, clearing overhead, blowing snow
  • windows and baffles: windows clear, barely any snow in the baffles

• 170803 05:02:22

  • cyclefridge enable=2, on all rx

• 170801 10:57:55 - 170803 05:02:22

  • /home/keck/sch/mce_autotune.sch
  • /data/keckdaq/autoqueue/pending/9_kCMB_50_202.sch
  • /home/keck/sch/runend.sch
  • weather: clear, hazy
  • windows and baffles: windows clear, very little snow in the baffles

• 170801 05:14:34

  • cyclefridge enable=2, on all rx

• 170730 10:42:10 - 170801 05:14:34

  • /home/keck/sch/mce_autotune.sch
  • /data/keckdaq/autoqueue/pending/9_kCMB_50_201.sch
  • massive He leak A high
  • due to do mechanical wear just behind the ladder
  • swapped to spare line
  • 170731 19:25:03 cycleFridge state=2, rx=rx0
  • cycle did not work, excessive heating, pumped on rx0
  • vacuum seemed good, cooled off very very slowly over the next hours
  • vacuum az line seems to have a leak as well
  • /home/keck/sch/runend.sch
  • weather: hazy, clear
  • windows and baffles: windows clear, very little snow in the baffles

• 170730 05:18:06

  • cyclefridge enable=2, on all rx
  • recharged comp 2

• 170728 10:50:52 - 170730 05:18:06

  • /home/keck/sch/mce_autotune.sch
  • /data/keckdaq/autoqueue/pending/9_kCMB_50_200.sch
  • az diff barley jumped over 30k
  • 170728 18:58:54 suspend_schedule
  • replaced 1A readhead roller
  • 170728 19:27:48 resume_schedule
  • /home/keck/sch/runend.sch
  • weather: clear
  • windows and baffles: windows clear, very little snow in the baffles

• 170728 05:25:57

  • cyclefridge enable=2, on all rx

• 170726 11:20:24 - 170728 05:25:57

  • /home/keck/sch/mce_autotune.sch
  • /data/keckdaq/autoqueue/pending/9_kCMB_50_199.sch
  • mediator stopped
  • restarted gcp, masd
  • 170726 18:14:05 Starting schedule: /home/keck/sch/mce_autotune.sch
  • 170726 18:17:15 Starting schedule: /data/keckdaq/autoqueue/pending/9_kCMB_50_199.sch
  • /home/keck/sch/runend.sch
  • weather: clear overhead, hazy, blowing snow, clear
  • windows and baffles: clear

• 170726 05:33:51

  • cyclefridge enable=2, on all rx

• 170724 11:11:54 - 170726 05:33:51

  • /home/keck/sch/mce_autotune.sch
  • /data/keckdaq/autoqueue/pending/9_kCMB_50_198.sch
  • /home/keck/sch/runend.sch
  • weather: clear, mostly clear overhead
  • windows and baffles: clear

• 170724 05:41:40

  • cyclefridge enable=2, on all rx

• 170722 11:21:43 - 170724 05:41:40

  • /home/keck/sch/mce_autotune.sch
  • /data/keckdaq/autoqueue/pending/9_kCMB_50_197.sch
  • /home/keck/sch/runend.sch
  • weather: hazy, blowing snow, clear
  • windows and baffles: clear

• 170722 05:49:34

  • cyclefridge enable=2, on all rx
  • recharged comp 3

• 170720 11:30:06 - 170722 05:49:34

  • /home/keck/sch/mce_autotune.sch
  • /data/keckdaq/autoqueue/pending/9_kCMB_50_196.sch
  • /home/keck/sch/runend.sch
  • weather: hazy, windy, very hazy
  • windows and baffles:

• 170720 05:57:26

  • cyclefridge enable=2, on all rx

• 170718 11:27:48 - 170720 05:57:26

  • /home/keck/sch/mce_autotune.sch
  • /data/keckdaq/autoqueue/pending/9_kCMB_50_195.sch
  • /home/keck/sch/runend.sch
  • weather: moslty clear, high winds blowing snow, hazy
  • windows and baffles: windows clear, very little snow on the baffles

• 170718 06:04:52

  • cyclefridge enable=2, on all rx
  • pmac stopped!??
  • el current drifting apart

• 170716 12:06:20 - 170718 06:04:52

  • /home/keck/sch/mce_autotune.sch
  • /data/keckdaq/autoqueue/pending/9_kCMB_50_194.sch
  • masd3 stopped, came back after reboot of keck36
  • /home/keck/sch/runend.sch
  • weather: hazy, high winds, blowing snow
  • windows and baffles: windows clear, little snow on the baffles

• 170716 06:13:10

  • cyclefridge enable=2, on all rx

• 170714 11:38:48 - 170716 06:13:10

  • /home/keck/sch/mce_autotune.sch
  • /data/keckdaq/autoqueue/pending/9_kCMB_50_193.sch
  • pmac stopped during sky dip
  • 170715 09:29:00 15min power outage
  • all rx warmed up a bit above 300mK, all rx cooled down again after power was restored
  • 170715 10:20:28 resume_schedule
  • el current drifting apart
  • /home/keck/sch/runend.sch
  • weather: very hazy, blowing snow, clearing, clear
  • windows and baffles: windows clear, very little snow in the baffles

• 170714 06:21:02

  • cyclefridge enable=2, on all rx
  • recharged comp 4,5

• 170712 11:53:46 - 170714 06:21:02

  • /home/keck/sch/mce_autotune.sch
  • /data/keckdaq/autoqueue/pending/9_kCMB_50_192.sch
  • mediator stopped
  • restarted gcp, masd
  • 70712 22:03:29 Starting schedule: /home/keck/sch/mce_autotune.sch
  • 170712 22:05:19 Starting schedule: /data/keckdaq/autoqueue/pending/9_kCMB_50_192.sch
  • /home/keck/sch/runend.sch
  • weather: mostly clear, very hazy, high winds, blowing snow
  • windows and baffles: windows clear, quite some snow in and on the baffles

• 170712 06:28:54

  • cyclefridge enable=2, on all rx
  • restarted gcp, masd
  • installed new spare hose

• 170710 12:02:44 - 170712 06:28:54

  • /home/keck/sch/mce_autotune.sch
  • /data/keckdaq/autoqueue/pending/9_kCMB_50_191.sch
  • mediator stopped
  • restarted gcp, masd
  • 170711 23:14:26 Starting schedule: /home/keck/sch/mce_autotune.sch
  • 170711 23:16:16 Starting schedule: /data/keckdaq/autoqueue/pending/9_kCMB_50_191.sch
  • on the last 3 scans of last scanset of the run, I played around with syncbox and disconnecting
  • one fibere to see if I could stall mediator, but did not
  • /home/keck/sch/runend.sch
  • weather: hazy, claering, very hazy, blowing snow, clear
  • windows and baffles: windows clear, some snow in and on the baffles

• 170710 06:36:47

  • cyclefridge enable=2, on all rx

• 170708 12:42:24 - 170710 06:36:47

  • /home/keck/sch/mce_autotune.sch
  • /data/keckdaq/autoqueue/pending/9_kCMB_50_190.sch
  • mediator stopped
  • restarted gcp, masd
  • 170709 15:14:14 Starting schedule: /home/keck/sch/mce_autotune.sch
  • 170709 15:16:05 Starting schedule: /data/keckdaq/autoqueue/pending/9_kCMB_50_190.sch
  • /home/keck/sch/runend.sch
  • weather: mostly clear, clear and cold, very hazy
  • windows and baffles: windows clear, some fresh thin snow on the baffles

• 170708 06:44:38

  • cyclefridge enable=2, on all rx
  • 170708 08:49:27 Starting schedule: /home/keck/sch/starpoint_pole_keck_winter.sch
  • 170708 12:15:26 Exiting schedule: /home/keck/sch/starpoint_pole_keck_winter.sch

• 170706 12:38:36 - 170708 06:44:38

  • /home/keck/sch/mce_autotune.sch
  • /data/keckdaq/autoqueue/pending/9_kCMB_50_189.sch
  • 170707 00:57:22 Mediator: Lost scanner connection to Mce0
  • restarted masd0 didn't help, cold restarted keck33
  • a few scansets without rx0
  • 170707 03:35:48 Mediator: Control connection established to Mce0
  • mediator stopped
  • restarted gcp, masd
  • 170707 14:47:09 Starting schedule: /home/keck/sch/mce_autotune.sch
  • 170707 14:48:58 Starting schedule: /data/keckdaq/autoqueue/pending/9_kCMB_50_189.sch
  • leak on rx2 lines
  • pressure was only down to 263psi on the high side, but rx2 reacts very strongly to pressure drops
  • recharged comp 1 and swaped for spare line, leaky line C high
  • small increase in FPU temperature, but recovered after recharge
  • /home/keck/sch/runend.sch
  • weather: hazy, blowing snow, clearing
  • windows and baffles: clear

• 170706 07:07:28

  • cyclefridge enable=2, on all rx

• 170704 12:32:00 - 170706 07:05:48

  • /home/keck/sch/mce_autotune.sch
  • /data/keckdaq/autoqueue/pending/9_kCMB_50_188.sch
  • mediator stopped
  • restarted gcp, masd
  • 170705 00:44:19 Starting schedule: /home/keck/sch/mce_autotune.sch
  • 170705 00:46:08 Starting schedule: /data/keckdaq/autoqueue/pending/9_kCMB_50_188.sch
  • mediator stopped
  • restarted gcp, masd
  • 170705 09:24:43 Starting schedule: /home/keck/sch/mce_autotune.sch
  • 170705 09:26:33 Starting schedule: /data/keckdaq/autoqueue/pending/9_kCMB_50_188.sch
  • /home/keck/sch/runend.sch
  • restarted gcp, masd
  • weather: overcast, hazy
  • windows and baffles: windows clear, snow on the baffles

• 170704 07:13:41

  • cyclefridge enable=2, on all rx
  • it was A low with the leak, replaced bad line with new spare

• 170702 12:57:08 - 170704 07:13:41

  • /home/keck/sch/mce_autotune.sch
  • /data/keckdaq/autoqueue/pending/9_kCMB_50_187.sch
  • /home/keck/sch/runend.sch
  • weather: hazy, very hazy, overcast
  • windows and baffles: windows clear, some snow on the baffles

• 170702 07:21:38

  • cyclefridge enable=2, on all rx
  • replaced A low with spare line
  • might be also A high, but with the small leak and both lines close to each other it is hard to tell
  • will monitor over night

• 170630 12:52:39 - 170702 07:21:38

  • /home/keck/sch/mce_autotune.sch
  • /data/keckdaq/autoqueue/pending/9_kCMB_50_186.sch
  • /home/keck/sch/runend.sch
  • weather: mostly clear overhead
  • windows and baffles: clear

• 170630 07:29:23

  • cyclefridge enable=2, on all rx
  • another He leak (small) on compressor 3, rx0

• 170628 13:00:58 - 170630 07:29:23

  • /home/keck/sch/mce_autotune.sch
  • /data/keckdaq/autoqueue/pending/9_kCMB_50_185.sch
  • /home/keck/sch/runend.sch
  • weather: very hazy, high winds, clearing, winds easing
  • windows and baffles: clear

• 170628 07:37:15

  • cyclefridge enable=2, on all rx
  • installed new spare He line
  • readjusted A1 back plane, was scratching slightly on one of the brass footings of the az track

• 170626 13:43:16 - 170628 07:37:15

  • /home/keck/sch/mce_autotune.sch
  • /data/keckdaq/autoqueue/pending/9_kCMB_50_184.sch
  • /home/keck/sch/runend.sch
  • weather: mostly clear overhead, blowing snow, hazy, high winds
  • windows and baffles: windows clear, little snow in and on the baffles

• 170626 07:45:06

  • cyclefridge enable=2, on all rx
  • replaced readhead B2
  • recharged comp 3,4 4 has a massive leak
  • swapped D Hi for spare

• 170624 13:05:23 - 170626 07:45:06

  • /home/keck/sch/mce_autotune.sch
  • problems with masd and mediator stopped
  • restarted gcp and masd
  • rebooted keck33-37, keck37 reported disk problems
  • raid array verified fine
  • masd4 won't start connection lost to control program
  • 170624 14:32:22 Starting schedule: /home/keck/sch/mce_autotune.sch
  • /data/keckdaq/autoqueue/pending/9_kCMB_50_183.sch
  • rx1 warmed up early, due to the problems during last fridgecycle
  • /home/keck/sch/runend1.sch
  • weather: hazy, high winds, blowing snow, mostly clear overhead
  • windows and baffles: windows clear, some snow in the baffles

• 170624 07:38:40

  • cyclefridge enable=2, on all rx
  • reseated all rx4 connectors and cards
  • antenna crashed
  • problems restarting the cycle
  • had to reboot keck31 and keck32
  • el motor current drifting apart
  • was about half way through cycle
  • 170624 09:44:44 did another single fridge cycle
  • spudws 2 unresponsive
  • restarted gcp and masd
  • implemented sptr webpage fix - works :)
  • changed schedlib.sch
  • implemented trans_TES fix - works as well :)
  • monitor plot for rx0 shows some strange gap after antenna crashed

• 170622 13:15:20 - 170624 07:38:40

  • /home/keck/sch/mce_autotune.sch
  • /data/keckdaq/autoqueue/pending/9_kCMB_50_182.sch
  • masd0 quit after the frist 2 scansets, had to wait to reboot until RAID rebuilding was finished
  • rebooted keck33, restarted masd0
  • looks like keck33 RAID is working again, still have to perform some tests
  • /home/keck/sch/runend.sch
  • weather: very hazy, hazy, windy
  • windows and baffles: windows clear, quite some snow in the baffles

• 170622 07:46:59

  • cyclefridge enable=2, on all rx
  • brief power outage around 1130 170622
  • restarted masd, rebuilding keck33 RAID array

• 170620 13:25:16 - 170622 07:46:59

  • /home/keck/sch/mce_autotune.sch
  • /data/keckdaq/autoqueue/pending/9_kCMB_50_181.sch
  • mediator stopped
  • 170621 13:27:15 Starting schedule: /home/keck/sch/mce_autotune.sch
  • 170621 13:28:38 Starting schedule: /data/keckdaq/autoqueue/pending/9_kCMB_50_181.sch
  • /home/keck/sch/runend.sch
  • weather: very hazy, warm
  • windows and baffles: windows clear, quite some snow in the baffles

• 170620 07:54:49

  • cyclefridge enable=2, on all rx
  • restarted masd

• 170618 13:27:35 - 170620 07:54:49

  • /home/keck/sch/mce_autotune.sch
  • /data/keckdaq/autoqueue/pending/9_kCMB_50_180.sch
  • mediator stopped
  • 170619 14:26:01 Starting schedule: /home/keck/sch/mce_autotune.sch
  • 170619 14:27:16 Starting schedule: /data/keckdaq/autoqueue/pending/9_kCMB_50_180.sch
  • /home/keck/sch/runend.sch
  • weather: clear, cold, getting overcast and warm, windy
  • windows and baffles: windows clear, some snow in the baffles in and on the baffles

• 170618 08:02:42

  • cyclefridge enable=2, on all rx
  • recharged comp 1 to 296psi, rx2 even pressure was still 285psi, seems to help with 4K temp
  • rx2 finished cycle again in normal time
  • restarted gcp, masd

• 170616 14:16:01 - 170618 08:02:42

  • /home/keck/sch/mce_autotune.sch
  • /data/keckdaq/autoqueue/pending/9_kCMB_50_179.sch
  • mediator stopped
  • 170617 12:53:30 Starting schedule: /home/keck/sch/startpager.sch
  • 170617 12:53:33 Starting schedule: /data/keckdaq/autoqueue/pending/9_kCMB_50_179.sch
  • /home/keck/sch/runend.sch
  • weather: hazy, clear, cold
  • windows and baffles: windows clear, very little snow in and on the baffles

• 170616 08:10:33

  • cyclefridge enable=2, on all rx
  • rx2 fridgecycle is taking longer than in the past

• 170614 14:23:28 - 170616 08:10:33

  • /home/keck/sch/mce_autotune.sch
  • /data/keckdaq/autoqueue/pending/9_kCMB_50_178.sch
  • /home/keck/sch/runend.sch
  • weather: hazy, overcast, hazy
  • windows and baffles: windows clear, some snow in and on the baffles

• 170614 08:18:24

  • cyclefridge enable=2, on all rx

• 170612 14:19:11 - 170614 08:18:24

  • /home/keck/sch/mce_autotune.sch
  • /data/keckdaq/autoqueue/pending/9_kCMB_50_177.sch
  • /home/keck/sch/runend.sch
  • weather: clear, hazy, very hazy
  • windows and baffles: windows clear, little snow on the baffles

• 170612 08:26:18

  • cyclefridge enable=2, on all rx

• 170610 14:12:35 - 170612 08:26:18

  • /home/keck/sch/mce_autotune.sch
  • /data/keckdaq/autoqueue/pending/9_kCMB_50_176.sch
  • mediator stopped
  • 170611 10:00:33 Starting schedule: /home/keck/sch/mce_autotune.sch
  • 170611 10:01:48 Starting schedule: /data/keckdaq/autoqueue/pending/9_kCMB_50_176.sch
  • /home/keck/sch/runend.sch
  • weather: bit hazy
  • windows and baffles: windows clear, some snow on the baffles

• 170610 08:34:08

  • cyclefridge enable=2, on all rx

• 170608 14:34:58 - 170610 08:34:08

  • /home/keck/sch/mce_autotune.sch
  • /data/keckdaq/autoqueue/pending/9_kCMB_50_175.sch
  • /home/keck/sch/runend.sch
  • weather: mostly clear, windy, haze
  • windows and baffles: windows clear, little snow on the baffles

• 170608 08:42:01

  • cyclefridge enable=2, on all rx
  • 170608 10:03:23 Starting schedule: /home/keck/sch/starpoint_pole_keck_winter.sch
  • 170608 13:57:18 Exiting schedule: /home/keck/sch/starpoint_pole_keck_winter.sch
  • recharged comp 2,3,4

• 170606 14:51:50 - 170608 08:42:01

  • /home/keck/sch/mce_autotune.sch
  • /data/keckdaq/autoqueue/pending/9_kCMB_50_174.sch
  • /home/keck/sch/runend.sch
  • weather: mostly clear, hazy, clear
  • windows and baffles: clear

• 170606 08:49:53

  • cyclefridge enable=2, on all rx
  • restarted gcp, masd

• 170604 17:23:36 - 170606 08:49:53

  • late start due to problems with cycle, see below
  • /home/keck/sch/mce_autotune.sch
  • /data/keckdaq/autoqueue/pending/9_kCMB_50_173.sch
  • mediator stopped
  • restarted gcp, masd
  • 170605 10:32:25 Starting schedule: /home/keck/sch/mce_autotune.sch
  • 170605 10:33:41 Starting schedule: /data/keckdaq/autoqueue/pending/9_kCMB_50_173.sch
  • /home/keck/sch/runend.sch
  • weather: mostly clear
  • windows and baffles: windows clear, very little snow in the baffles

• 170604 08:58:46

  • cyclefridge enable=2, on all rx
  • 4K warming up abnormal cycle
  • stopped cycle
  • problems with antenna, rebooted keck31
  • waiting on 4K to come down again
  • 170604 11:06:03 Starting schedule: /home/keck/sch/fridgecycle.sch
  • still problems with keck33 RAID controller
  • booted computer with HDD directly connected to Mainboard, bypassing RAID controller

• 170602 15:41:08 - 170604 08:58:46

  • /home/keck/sch/mce_autotune.sch
  • /data/keckdaq/autoqueue/pending/9_kCMB_50_172.sch
  • masd0 stopped
  • restarted masd
  • 170602 15:46:56 Starting schedule: /home/keck/sch/mce_autotune.sch
  • 170602 15:48:11 Starting schedule: /data/keckdaq/autoqueue/pending/9_kCMB_50_172.sch
  • mediator stopped AGAIN!!!
  • keck33 is unresponsive
  • problems with keck33 after the rebuilding the array
  • starting the run without keck33
  • 170603 02:05:55 Starting schedule: /home/keck/sch/mce_autotune.sch
  • 170603 02:07:10 Starting schedule: /data/keckdaq/autoqueue/pending/9_kCMB_50_172.sch
  • /home/keck/sch/runend.sch
  • weather: mostly clear,hazy,
  • windows and baffles: windows clear, quite some snow in the baffles

• 170602 09:48:52

  • cyclefridge enable=2, on all rx
  • worked on RAID array on keck32, webinterface is working
  • keck33 also had one bad disk, rebuilding RAID array
  • restarted keck32
  • restarted gcp, masd

• 170531 14:58:27 - 170602 09:48:52

  • /home/keck/sch/mce_autotune.sch
  • /data/keckdaq/autoqueue/pending/9_kCMB_50_171.sch
  • mediator stopped
  • 170601 12:08:50 Starting schedule: /home/keck/sch/mce_autotune.sch
  • 170601 12:10:05 Starting schedule: /data/keckdaq/autoqueue/pending/9_kCMB_50_171.sch
  • mediator stopped
  • aborted run, was after 2300LST
  • /home/keck/sch/runend.sch
  • weather: mostly clear
  • windows and baffles: clear

• 170531 09:13:28

  • cyclefridge enable=2, on all rx

• 170529 15:12:30 - 170531 09:13:28

  • /home/keck/sch/mce_autotune.sch
  • /data/keckdaq/autoqueue/pending/9_kCMB_50_170.sch
  • mediator stopped
  • restarted gcp, masd
  • 170530 10:57:18 Starting schedule: /home/keck/sch/mce_autotune.sch
  • 170530 10:58:33 Starting schedule: /data/keckdaq/autoqueue/pending/9_kCMB_50_170.sch
  • /home/keck/sch/runend.sch
  • weather: mostly clear
  • windows and baffles: clear

• 170529 09:21:19

  • cyclefridge enable=2, on all rx
  • el current drifting apart

• 170527 15:01:09 - 170529 09:21:19

  • /home/keck/sch/mce_autotune.sch
  • /data/keckdaq/autoqueue/pending/9_kCMB_50_169.sch
  • /home/keck/sch/runend.sch
  • weather: mostly clear
  • windows and baffles: clear

• 170527 09:29:12

  • cyclefridge enable=2, on all rx

• 170525 15:08:28 - 170527 09:29:12

  • /home/keck/sch/mce_autotune.sch
  • /data/keckdaq/autoqueue/pending/9_kCMB_50_168.sch
  • /home/keck/sch/runend.sch
  • weather: mostly clear, hazy
  • windows and baffles: windows clear, very little snow in the baffles

• 170525 09:37:02

  • cyclefridge enable=2, on all rx
  • put new spare He line in
  • replaced A high with spare
  • put new spare He line in
  • el current drifting apart

• 170523 15:19:25 - 170525 09:37:02

  • /home/keck/sch/mce_autotune.sch
  • /data/keckdaq/autoqueue/pending/9_kCMB_50_167.sch
  • massive He leak in A high, had to recharge twice during run
  • around 170525 0400 rx0 FPU went a bit over 300mK
  • recovered after recharge
  • /home/keck/sch/runend.sch
  • weather: clear, clear overhead
  • windows and baffles: clear

• 170523 09:44:55

  • cyclefridge enable=2, on all rx
  • swapped Raid controller in keck32 with spare

• 170521 15:31:53 - 170523 09:44:55

  • /home/keck/sch/mce_autotune.sch
  • /data/keckdaq/autoqueue/pending/9_kCMB_50_166.sch
  • /home/keck/sch/runend.sch
  • weather: mostly clear, wind picking up again
  • windows and baffles: clear

• 170521 09:52:47

  • cyclefridge enable=2, on all rx
  • recharged comp 2, swapped B high with spare line

• 170519 15:37:54 - 170521 09:52:47

  • /home/keck/sch/mce_autotune.sch
  • /data/keckdaq/autoqueue/pending/1_test_plc_at_diff_fpu_temps.sch
  • 170519 15:41:24 abort_schedule
  • lost connection to masd 0,1,3
  • restarted masd
  • 170519 15:43:12 Starting schedule: /home/keck/sch/mce_autotune.sch
  • 170519 15:44:40 Starting schedule: /data/keckdaq/autoqueue/pending/1_test_plc_at_diff_fpu_temps.sch
  • 170519 17:48:52 Starting schedule: /data/keckdaq/autoqueue/pending/9_kCMB_50_165.sch
  • /home/keck/sch/runend.sch
  • weather: mostly clear, hazy, blowing snow, clear overhead
  • windows and baffles:

• 170519 10:00:46

  • cyclefridge enable=2, on all rx
  • recharged comp 3,4

• 170517 16:26:04 - 170519 10:00:46

  • /home/keck/sch/mce_autotune.sch
  • /data/keckdaq/autoqueue/pending/9_kCMB_50_164.sch
  • small az-diff jump just over 30k
  • 170518 00:57:01 suspend_schedule
  • replaced readhead roller 1B
  • data would be good through the jump, since it was only barely above 30k but definitely after resume schedule
  • 170518 01:41:24 resume_schedule
  • 170518 07:36 archiving stopped
  • Disk suddenly on read only mode
  • restarted keck32
  • mediator stopped after few seconds, most likely an old open archive file, closed it by hand
  • 170518 08:38:11 file 20170518_072445.dat.gz most likely corrupt
  • 170518 08:37:13 Starting schedule: /home/keck/sch/mce_autotune.sch
  • 170518 08:38:27 Starting schedule: /data/keckdaq/autoqueue/pending/9_kCMB_50_164.sch
  • /home/keck/sch/runend.sch
  • weather: hazy, high winds, blowing snow
  • windows and baffles: windows clear, some snow in and on the baffles

• 170517 10:46:31

  • cyclefridge enable=2, on all rx

• 170515 17:05:10 - 170517 10:46:31

  • later start due to keck32 problems
  • /home/keck/sch/mce_autotune.sch
  • /data/keckdaq/autoqueue/pending/9_kCMB_50_163.sch
  • 170516 02:14:16 suspend_schedule
  • stopped run to attend puls tube lines in rx4, only way to get to it is from above
  • lifted the frontplate a bit, luckely rx4 is where we the pinion gear is so there were cuts
  • didn't see any lines touching, but only limited view, put some absorbent pads between lines and rx4
  • that is all we can do now
  • 170516 02:41:30 abort_schedule
  • so much snow in the baffles in the last 12h so I had to abort cmb run and run the snow schedule
  • since it was warm and not much wind, we used the time to clear out snow from the ground shield from the last days
  • with 2 of us it took us close to 2h, tons of snow
  • 170516 04:27:20 Starting schedule: /home/keck/sch/mce_autotune.sch
  • 170516 04:28:35 Starting schedule: /data/keckdaq/autoqueue/pending/9_kCMB_50_163.sch
  • 170516 07:04:43 el motor current drifting apart
  • reset el motor current
  • 170517 02:57:11 abort_schedule
  • working more on debugging mediator stops with Grantland
  • 170517 04:42:04 Starting schedule: /home/keck/sch/mce_autotune.sch
  • 170517 04:43:18 Starting schedule: /data/keckdaq/autoqueue/pending/9_kCMB_50_163.sch
  • /home/keck/sch/runend.sch
  • weather: very hazy, thin clouds
  • windows and baffles: windows clear, quite some snow in and on the baffles

• 170515 10:49:11

  • cyclefridge enable=2, on all rx
  • recharged comp 2
  • IT did some changes to the spfs, therefore there were some problems with data going out
  • changes sptr_put.pl accordingly and data are moved to spfs again

• 170513 16:04:22 - 170515 08:30

  • /home/keck/sch/mce_autotune.sch
  • /data/keckdaq/autoqueue/pending/9_kCMB_50_162.sch
  • keck 32 crashed
  • when rebooting keck 32, it showed degraded RAID array
  • removed faulty disk and installed new one and started rebuilding RAID array
  • weather: hazy, very hazy
  • windows and baffles: windows clear, some snow in the baffles

• 170513 10:24:14

  • cyclefridge enable=2, on all rx

• 170511 15:53:56 - 170513 10:24:14

  • /home/keck/sch/mce_autotune.sch
  • /data/keckdaq/autoqueue/pending/9_kCMB_50_161.sch
  • /home/keck/sch/runend.sch
  • weather: hazy, warm
  • windows and baffles: windows clear, some snow in the baffles

• 170511 10:32:06

  • cyclefridge enable=2, on all rx

• 170509 16:04:35 - 170511 10:32:06

  • /home/keck/sch/mce_autotune.sch
  • /data/keckdaq/autoqueue/pending/9_kCMB_50_160.sch
  • /home/keck/sch/runend.sch
  • weather: clear, totally overcast, wind picking up
  • windows and baffles: windows clear, some snow in on the baffles

• 170509 10:39:57

  • cyclefridge enable=2, on all rx
  • recharged comp 2
  • restarted gcp, masd

• 170507 16:23:29 - 170509 10:39:57

  • /home/keck/sch/mce_autotune.sch
  • /data/keckdaq/autoqueue/pending/9_kCMB_50_159.sch
  • omega0 eth0 stopped again, restarted network for now, will observe
  • /home/keck/sch/runend.sch
  • weather: little hazy, claering, clear and cold
  • windows and baffles: windows clear, some snow on the baffles

• 170507 10:53:08

  • cyclefridge enable=2, on all rx

• 170505 16:39:14 - 170507 10:50:40

  • /home/keck/sch/mce_autotune.sch
  • /data/keckdaq/autoqueue/pending/9_kCMB_50_158.sch
  • az diff jump
  • 170506 03:32:31 resume_schedule
  • date a good before jump and after resume
  • az diff jump, replaced readhead roller 3A
  • 170506 13:25:09 resume_schedule
  • date a good before jump and after resume
  • omega0 network problems to station, lost both keckviewers in the station
  • restart keckviewer in station
  • lost connection again
  • could not reconnect ot omega0 ???
  • mediator stopped
  • 170507 10:50:40 abort_schedule
  • weather: clear, very hazy
  • windows and baffles: windows clear, quite some snow in the baffles

• 170505 11:00:09

  • cyclefridge enable=2, on all rx
  • 170505 12:03:51 Starting schedule: /home/keck/sch/starpoint_pole_keck_winter.sch
  • 170505 16:38:23 Exiting schedule: /home/keck/sch/starpoint_pole_keck_winter.sch

• 170503 16:29:47 - 170505 10:55:40

  • /home/keck/sch/mce_autotune.sch
  • /data/keckdaq/autoqueue/pending/9_kCMB_50_157.sch
  • /home/keck/sch/runend.sch
  • restarted gcp and masd
  • weather: hazy, clearing, clear
  • windows and baffles: windows clear, barely any snow on the baffels

• 170503 11:03:33

  • cyclefridge enable=2, on all rx
  • recharged comp 2,3
  • starting a debug version (for mediator) of gcp
  • restarted gcp and masd

• 170501 16:28:26 - 170503 11:03:33

  • /home/keck/sch/mce_autotune.sch
  • /data/keckdaq/autoqueue/pending/9_kCMB_50_156.sch
  • mediator stopped
  • 170503 07:04:32 Starting schedule: /home/keck/sch/mce_autotune.sch
  • 170503 07:05:46 Starting schedule: /data/keckdaq/autoqueue/pending/9_kCMB_50_156.sch
  • /home/keck/sch/runend.sch
  • weather: clear, cold, hazy towards runend
  • windows and baffles: windows clear, very little snow on the baffles

• 170501 11:11:23

  • cyclefridge enable=2, on all rx
  • restarted masd

• 170429 16:41:15 - 170501 11:11:23

  • /home/keck/sch/mce_autotune.sch
  • /data/keckdaq/autoqueue/pending/9_kCMB_50_155.sch
  • /home/keck/sch/runend.sch
  • weather: clear, mostly clear, cold
  • windows and baffles: windows clear, very little snow on the baffles

• 170429 11:19:17

  • cyclefridge enable=2, on all rx
  • restarted gcp, masd to initiate a some debugging changes for mediator
  • recharged comp 4

• 170427 17:20:03 - 170429 11:19:17

  • /home/keck/sch/mce_autotune.sch
  • /data/keckdaq/autoqueue/pending/9_kCMB_50_154.sch
  • /home/keck/sch/runend.sch
  • weather: hazy, clearing, some haze, clear towards runend
  • windows and baffles: windows clear, very little snow on the baffles

• 170427 11:27:08

  • cyclefridge enable=2, on all rx
  • el current drift
  • recharged comp 2

• 170425 16:54:05 - 170427 11:27:08

  • /home/keck/sch/mce_autotune.sch
  • /data/keckdaq/autoqueue/pending/9_kCMB_50_153.sch
  • mediator stopped
  • restarted gcp, masd
  • problems with masd2, restarted keck35
  • 170426 05:49:09 Starting schedule: /home/keck/sch/mce_autotune.sch
  • 170426 05:50:25 Starting schedule: /data/keckdaq/autoqueue/pending/9_kCMB_50_153.sch
  • weather is not good so decided to stop the run briefly to put in new spare He line
  • 170427 04:46:13 suspend_schedule
  • 170427 05:14:36 resume_schedule
  • mediator stopped
  • restarted gcp, masd
  • 170427 05:52:58 Starting schedule: /home/keck/sch/mce_autotune.sch
  • 170427 05:54:12 Starting schedule: /data/keckdaq/autoqueue/pending/9_kCMB_50_153.sch
  • /home/keck/sch/runend.sch
  • weather: overcast, very hazy
  • windows and baffles: windows clear, quite some snow in and on the baffles

• 170425 11:35:02

  • cyclefridge enable=2, on all rx
  • restarted gcp, masd

• 170423 17:01:10 - 170425 11:35:02

  • /home/keck/sch/mce_autotune.sch
  • /data/keckdaq/autoqueue/pending/9_kCMB_50_152.sch
  • mediator stopped
  • restarted gcp, masd
  • 170425 06:46:31 Starting schedule: /home/keck/sch/mce_autotune.sch
  • 170425 06:47:46 Starting schedule: /data/keckdaq/autoqueue/pending/9_kCMB_50_152.sch
  • /home/keck/sch/runend.sch
  • weather: cold and clear, warming up, getting hazy, overcast
  • windows and baffles: windows little snow in the center, some snow on the baffles

• 170423 11:42:53

  • cyclefridge enable=2, on all rx

• 170421 17:19:09 - 170423 11:42:53

  • /home/keck/sch/mce_autotune.sch
  • /data/keckdaq/autoqueue/pending/1_knoise_responsivity_9_022.sch
  • 170421 23:26:07 Starting schedule: /data/keckdaq/autoqueue/pending/9_kCMB_50_151.sch
  • /home/keck/sch/runend.sch
  • weather: cold and clear
  • windows and baffles: clear

• 170421 11:50:51

  • cyclefridge enable=2, on all rx
  • recharged comp 2

• 170419 17:15:50 - 170421 11:50:51

  • /home/keck/sch/mce_autotune.sch
  • /data/keckdaq/autoqueue/pending/9_kCMB_50_150.sch
  • /home/keck/sch/runend.sch
  • weather: clear, cold
  • windows and baffles: clear

• 170419 11:58:36

  • cyclefridge enable=2, on all rx
  • recharged comp 3,5 leak on comp 5
  • replaced E Low with spare

• 170417 17:29:10 - 170419 11:58:36

  • /home/keck/sch/mce_autotune.sch
  • /data/keckdaq/autoqueue/pending/9_kCMB_50_149.sch
  • mediator stopped
  • 170418 04:50:29 Starting schedule: /home/keck/sch/mce_autotune.sch
  • 170418 04:51:44 Starting schedule: /data/keckdaq/autoqueue/pending/9_kCMB_50_149.sch
  • /home/keck/sch/runend.sch
  • weather: clear, cold
  • windows and baffles: clear

• 170417 12:06:34

  • cyclefridge enable=2, on all rx

• 170415 17:38:07 - 170417 12:06:34

  • /home/keck/sch/mce_autotune.sch
  • /data/keckdaq/autoqueue/pending/9_kCMB_50_148.sch
  • /home/keck/sch/runend.sch
  • weather: clear
  • windows and baffles: windows clear, some snow in the baffles

• 170415 12:14:20

  • cyclefridge enable=2, on all rx
  • 170415 13:53:25 Starting schedule: /home/keck/sch/starpoint_pole_keck_winter.sch
  • 170415 17:38:07 Exiting schedule: /home/keck/sch/starpoint_pole_keck_winter.sch
  • el current drifiting apart

• 170413 18:13:34 - 170415 12:14:20

  • /home/keck/sch/mce_autotune.sch
  • /data/keckdaq/autoqueue/pending/9_kCMB_50_147.sch
  • /home/keck/sch/runend.sch
  • weather: thin clouds, haze, clear, getting cold
  • windows and baffles: windows clear, some snow on the baffles

• 170413 12:36:04

  • cold reboot of keck32 and keck31
  • restarted gcp and masd
  • cyclefridge enable=2, on all rx
  • webplots stopped
  • problems with webplots & control, had to restart keck32 again
  • restarted gcp and masd

• 170411 17:52:14 - 170413 12:22:11

  • /home/keck/sch/mce_autotune.sch
  • /data/keckdaq/autoqueue/pending/9_kCMB_50_146.sch
  • mediator stopped
  • restarted gcp and masd
  • 170413 00:23:45 Starting schedule: /home/keck/sch/mce_autotune.sch
  • 170413 00:34:29 Starting schedule: /data/keckdaq/autoqueue/pending/9_kCMB_50_146.sch
  • gcp was in a wired state
  • restarted gcp and masd
  • 170413 03:52:19 Starting schedule: /home/keck/sch/mce_autotune.sch
  • 170413 03:53:32 Starting schedule: /data/keckdaq/autoqueue/pending/9_kCMB_50_146.sch
  • /home/keck/sch/runend.sch
  • weather: haze, partly clear, overcast, haze
  • windows and baffles: windows clear, some snow in and on the baffles

• 170411 12:30:02

  • cyclefridge enable=2, on all rx

• 170409 17:55:14 - 170411 12:30:02

  • /home/keck/sch/mce_autotune.sch
  • /data/keckdaq/autoqueue/pending/9_kCMB_50_145.sch
  • mediator stopped
  • restarted gcp and masd
  • 170411 04:34:53 Starting schedule: /home/keck/sch/mce_autotune.sch
  • 170411 04:36:08 Starting schedule: /data/keckdaq/autoqueue/pending/9_kCMB_50_145.sch
  • /home/keck/sch/runend.sch
  • weather: totally overcast, clear
  • windows and baffles: windows clear, very little snow in the baffles

• 170409 12:37:53

  • cyclefridge enable=2, on all rx
  • recharged comp 4

• 170407 18:38:41 - 170409 12:37:53

  • /home/keck/sch/mce_autotune.sch
  • /data/keckdaq/autoqueue/pending/9_kCMB_50_144.sch
  • /home/keck/sch/runend.sch
  • weather: haze, clear, totally overcast
  • windows and baffles: windows clear, quite some snow in and on baffles

• 170407 12:45:46

  • cyclefridge enable=2, on all rx

• 170405 18:29:40 - 170407 12:45:46

  • /home/keck/sch/mce_autotune.sch
  • /data/keckdaq/autoqueue/pending/9_kCMB_50_143.sch
  • mediator stopped
  • restarted gcp, masd
  • 170406 03:22:36 Starting schedule: /home/keck/sch/mce_autotune.sch
  • 170406 03:24:17 Starting schedule: /data/keckdaq/autoqueue/pending/9_kCMB_50_143.sch
  • /home/keck/sch/runend.sch
  • weather: blowing snow, hazy, clear, some clouds
  • windows and baffles: clear

• 170405 12:54:07

  • cyclefridge enable=2, on all rx
  • recharged comp 2
  • reset sync box
  • restarted gcp and masd

• 170403 21:07:22 - 170405 12:54:07

  • /home/keck/sch/mce_autotune.sch
  • /data/keckdaq/autoqueue/pending/9_kCMB_50_142.sch
  • mediator stopped again
  • restarted gcp, masd
  • 170405 08:12:46 Starting schedule: /home/keck/sch/mce_autotune.sch
  • 170405 08:14:00 Starting schedule: /data/keckdaq/autoqueue/pending/9_kCMB_50_142.sch
  • 170405 12:51:29 abort_schedule
  • /home/keck/sch/runend.sch
  • weather: thin clouds, overcast, blowing snow, high winds, hazy
  • windows and baffles: windows clear, lots of snow in and on the baffles

• 170403 13:04:55

  • cyclefridge enable=2, on all rx
  • recharged comp 3
  • replaced 4B readhead roller
  • while reading mediator, i.e. marking the text to with mouse to read it, mediator stopped, rx1,2,3 didn't finish cycle
  • 170403 17:54:15 cycleFridge enable=1 on all rx

• 170401 18:40:29 - 170403 13:04:55

  • /home/keck/sch/mce_autotune.sch
  • /data/keckdaq/autoqueue/pending/9_kCMB_50_141.sch
  • no siginficant heating on rx1,3,4
  • 170401 18:44:10 abort_schedule
  • 170401 18:45:39 Starting schedule: /home/keck/sch/mce_autotune.sch
  • 170401 18:46:54 Starting schedule: /data/keckdaq/autoqueue/pending/9_kCMB_50_141.sch
  • mediator stopped again
  • had to restart gcp, masd
  • 170403 04:37:01 Starting schedule: /home/keck/sch/mce_autotune.sch
  • 170403 04:38:57 Starting schedule: /data/keckdaq/autoqueue/pending/9_kCMB_50_141.sch
  • 170403 13:02:19 abort_schedule
  • /home/keck/sch/runend.sch
  • weather: haze, clear, thin clouds
  • windows and baffles: windows clear, some snow in and on baffles

• 170401 13:09:23

  • cyclefridge enable=2, on all rx

• 170330 18:48:34 - 170401 13:09:23

  • /home/keck/sch/mce_autotune.sch
  • /data/keckdaq/autoqueue/pending/9_kCMB_50_140.sch
  • /home/keck/sch/runend.sch
  • weather: overcast, thin clouds
  • windows and baffles: windows clear, some snow in the baffles

• 170330 13:28:55

  • restarted gcp,masd
  • cyclefridge enable=2, on all rx

• 170328 18:31:43 - 170330 13:17:15

  • /home/keck/sch/mce_autotune.sch
  • /data/keckdaq/autoqueue/pending/9_kCMB_50_139.sch
  • mediator stopped
  • restarted gcp, masd
  • 170329 06:25:06 Starting schedule: /home/keck/sch/mce_autotune.sch
  • 170329 06:28:08 Starting schedule: /data/keckdaq/autoqueue/pending/9_kCMB_50_139.sch
  • /home/keck/sch/runend.sch
  • weather: clear, getting overcast
  • windows and baffles: windows clear, quite some snow in and on the baffles

• 170328 15:26:22

  • cyclefridge enable=1, on all rx
  • recharged comp 2

• 170328 09:13:18 - 170328 15:24:23

  • /home/keck/sch/mce_autotune.sch
  • /data/keckdaq/autoqueue/pending/1_knoise_responsivity_9_021.sch
  • weather: clear overhead

• 170328 03:54:38

  • cyclefridge enable=2, on all rx

• 170326 18:25:31 - 170328 03:44:20

  • /home/keck/sch/mce_autotune.sch
  • /data/keckdaq/autoqueue/pending/9_kCMB_50_138.sch
  • saw no significant heatin on rx1,3
  • restarted gcp & masd
  • 170326 18:30:36 Starting schedule: /home/keck/sch/mce_autotune.sch
  • 170326 18:31:50 Starting schedule: /data/keckdaq/autoqueue/pending/9_kCMB_50_138.sch
  • mediator stopped
  • restarted gcp, masd
  • rx1 was at .5K UC as well, decided on fridgecycle
  • weather: clear, some clouds
  • windows and baffles:

• 170326 13:32:59

  • cyclefridge enable=2, on all rx
  • 170326 14:54:43 Starting schedule: /home/keck/sch/starpoint_pole_keck_summer.sch
  • 170326 16:59:19 Starting schedule: /home/keck/sch/starpoint_pole_keck_summer.sch

• 170324 19:07:00 - 170326 13:32:59

  • /home/keck/sch/mce_autotune.sch
  • /data/keckdaq/autoqueue/pending/9_kCMB_50_137.sch
  • don't see very much fpu heating on rx1,3,4
  • 170324 19:11:45 abort_schedule
  • restarted masd and gcp
  • 170324 19:13:30 Starting schedule: /home/keck/sch/mce_autotune.sch
  • 170324 19:14:44 Starting schedule: /data/keckdaq/autoqueue/pending/9_kCMB_50_137.sch
  • see fpu heating except rx3
  • /home/keck/sch/runend.sch
  • weather: clear overhead
  • windows and baffles: windows clear, very little snow in the baffles

• 170324 13:40:49

  • cyclefridge enable=2, on all rx

• 170322 18:56:11 - 170324 13:40:49

  • /home/keck/sch/mce_autotune.sch
  • /data/keckdaq/autoqueue/pending/9_kCMB_50_136.sch
  • /home/keck/sch/runend.sch
  • weather: clear overhead
  • windows and baffles: windows clear, very little snow in the baffles

• 170322 10:33:07

  • cyclefridge enable=2, on all rx
  • recharged comp4
  • rx4 tcm heater tuning
  • adjusted target value to -2.10

• 170320 19:44:45 - 170322 10:33:07

  • /home/keck/sch/mce_autotune.sch
  • /data/keckdaq/autoqueue/pending/9_kCMB_50_135.sch
  • 170320 19:54:54 abort_schedule
  • did not see trans_TES on rx1,3,4
  • restarted masd
  • 170320 19:56:00 Starting schedule: /home/keck/sch/mce_autotune.sch
  • 170320 19:57:14 Starting schedule: /data/keckdaq/autoqueue/pending/9_kCMB_50_135.sch
  • still not seeing the spikes very well!?
  • 170322 10:30:31 abort_schedule, to have more time to tune rx4 after cycle
  • /home/keck/sch/runend.sch
  • weather: overcast, blowing snow, clearing
  • windows and baffles: windows clear, lots of snow in and on baffles

• 170320 13:56:21

  • cyclefridge enable=2, on all rx
  • el current drifting apart

• 170318 16:42:24 - 170320 13:56:21

  • /home/keck/sch/mce_autotune.sch
  • 170318 16:57:28 Starting schedule: /data/keckdaq/autoqueue/pending/1_superfastnoise_1_008.sch
  • schedule stopped after a while
  • MAS daemon is not running or not responding
  • 170319 00:45:24 abort_schedule
  • restarted masd
  • 170319 00:50:59 Starting schedule: /home/keck/sch/mce_autotune.sch
  • 170319 00:51:33 Starting schedule: /data/keckdaq/autoqueue/pending/9_kCMB_50_134.sch
  • mediator stopped again!!!
  • restarted gcp, masd
  • 170319 01:02:28 Starting schedule: /home/keck/sch/mce_autotune.sch
  • 170319 01:03:42 Starting schedule: /data/keckdaq/autoqueue/pending/9_kCMB_50_134.sch
  • 170319 01:04:40 Mediator: Lost connection to Mce3
  • 170319 01:04:41 Mediator: Lost connection to Mce1
  • 170319 01:06:04 abort_schedule
  • 170319 01:08:20 Starting schedule: /home/keck/sch/mce_autotune.sch
  • 170319 01:09:36 Starting schedule: /data/keckdaq/autoqueue/pending/9_kCMB_50_134.sch
  • /home/keck/sch/runend.sch
  • weather: overcast, blowing snow, clear period, getting overcast again
  • windows and baffles: windows clear, quite some snow in and on the baffles

• 170318 11:14:38

  • cyclefridge enable=2, on all rx

• 170316 19:24:16 - 170318 08:17:07

  • /home/keck/sch/mce_autotune.sch
  • /data/keckdaq/autoqueue/pending/9_kCMB_50_133.sch
  • az diff jump, replaced readhead roller A3, readjusted B3
  • readjusted osborne rollers on wheel housing 3
  • mediator crashed again
  • cold reboot of keck31 and 32
  • el motor current drifting apart
  • 170318 08:17:07 abort_schedule
  • possible He leak in compressor 5
  • He leak E Hi afrimative, replaced az line with spare
  • el motor current drifting apart
  • can't write log files, wired state of controlSystem
  • had to restartkeck31 and keck32 again
  • weather: overcast, wind increasing, totally overcast, high winds, low vis
  • windows and baffles: a bit of snow on the windows, quite some snow in and on the baffles

• 170316 11:46:26

  • cyclefridge enable=2, on all rx

• 170316 03:24:49

  • cyclefridge enable=2, on all rx
  • recharged compressor 3,5

• 170315 06:12:39 - 170316 02:05:45

  • /home/keck/sch/mce_autotune.sch
  • /data/keckdaq/autoqueue/pending/9_kCMB_50_132.sch
  • mediator stopped around 170316 01:20
  • restarted mediator, but still some errors
  • had to restart gcp
  • /home/keck/sch/runend.sch
  • weather: mostly clear, getting overcast
  • windows and baffles: windows clear, some snow in and on baffles

• 170314 15:58:01

  • cyclefridge enable=2, on all rx

• 170314 02:51:48 - 170314 15:58:01

  • /home/keck/sch/mce_autotune.sch
  • /data/keckdaq/autoqueue/pending/1_knoise_responsivity_9_019.sch
  • masd3 stopped, restarted masd3 strange scroll of masd3 in keckviewer?
  • 170314 03:25:50 abort_schedule
  • 170314 03:26:24 Starting schedule: /data/keckdaq/autoqueue/pending/1_knoise_responsivity_9_019.sch
  • some masd problems, masd no responding
  • 170314 04:03:38 Starting schedule: /data/keckdaq/autoqueue/pending/1_knoise_responsivity_9_019.sch
  • 170314 10:00:10 Exiting schedule: /data/keckdaq/autoqueue/pending/1_knoise_responsivity_9_019.sch
  • 170314 10:00:10 Starting schedule: /data/keckdaq/autoqueue/pending/1_knoise_responsivity_9_020.sch
  • 170314 15:58:01 Exiting schedule: /data/keckdaq/autoqueue/pending/1_knoise_responsivity_9_020.sch
  • /home/keck/sch/runend.sch
  • weather: clear and cold during run 019, some thin clouds during 020
  • windows and baffles: clear

• 170313 19:52:23

  • cyclefridge enable=2, on all rx
  • some problems with the N2 purge system, we use way to much N2

• 170313 07:40:26 - 170313 19:47:31

  • mediator crashed
  • weather: clear
  • windows and baffles: clear

• 170313 01:39:05 - 170313 07:40:26

  • /home/keck/sch/mce_autotune.sch
  • /data/keckdaq/autoqueue/pending/1_knoise_responsivity_9_017.sch
  • el motor currents drifting apart
  • 170313 04:07:44 restarted masd3
  • weather: clear, some very thin clouds

---

CMB 2017

---

• 170312 18:18:49

  • cyclefridge enable=2, on all rx

• 170312 04:42:47

  • cyclefridge enable=2, on all rx

• 170311 22:34:24

  • cyclefridge enable=2, on rx0

• 170311 09:55:10

  • cyclefridge enable=2, on all rx
  • rx0 still gives problems during the cycle, 4K warms up over 10K with state 9 in the cycle???
  • stopped cycle on rx0 and set all heaters to 0
  • turns out rx0 always goes qutie high, just about to 10K
  • so cycles are good

• 170311 05:22:33

  • 170310 21:55 changed data compression back to lzma, all fff data went out via satellite
  • removed FFF, only minimal warm up, compressors were off for less than 15min. rx0 needed a bit of pumping
  • cyclefridge enable=2, on all rx
  • 170311 05:31:09 Starting schedule: /home/keck/sch/starpoint_pole_keck_summer.sch
  • still some problems with fridge cycle on rx0

• 170310 09:05:27

  • cyclefridge enable=2, on all rx

• 170310 01:23:46 - 170310 09:05:26

  • /home/keck/sch/mce_autotune.sch
  • /data/keckdaq/autoqueue/pending/7_ffflat_dslmast_33j_008.sch

• 170309 20:02:12

  • cyclefridge enable=2, on all rx

• 170309 12:20:42 - 170309 20:02:12

  • /home/keck/sch/mce_autotune.sch
  • /data/keckdaq/autoqueue/pending/7_ffflat_dslmast_33i_008.sch

• 170309 07:14:08

  • cyclefridge enable=2, on all rx

• 170308 23:32:46 - 170309 07:14:08

  • /home/keck/sch/mce_autotune.sch
  • /data/keckdaq/autoqueue/pending/7_ffflat_dslmast_33e_008.sch

• 170308 18:30:38

  • cyclefridge enable=2, on all rx

• 170308 10:48:51 - 170308 18:30:38

  • /home/keck/sch/mce_autotune.sch
  • /data/keckdaq/autoqueue/pending/7_ffflat_dslmast_33j_007.sch

• 170308 04:58:14

  • cyclefridge enable=2, on all rx

• 170307 21:18:06 - 170308 04:58:14

  • /home/keck/sch/mce_autotune.sch
  • /data/keckdaq/autoqueue/pending/7_ffflat_dslmast_33i_007.sch

• 170307 14:48:41

  • cyclefridge enable=2, on all rx

• 170307 07:05:00 - 170307 14:48:41

  • /home/keck/sch/mce_autotune.sch
  • /data/keckdaq/autoqueue/pending/7_ffflat_dslmast_33h_007.sch

• 170307 01:41:22

  • cyclefridge enable=2, on all rx

• 170306 18:00:00 - 170307 01:41:22

  • /home/keck/sch/mce_autotune.sch
  • /data/keckdaq/autoqueue/pending/7_ffflat_dslmast_33g_007.sch

• 170306 11:43:08

  • cyclefridge enable=2, on all rx
  • recharged comp 2,3

• 170306 04:01:28 - 170306 11:43:08

  • /home/keck/sch/mce_autotune.sch
  • /data/keckdaq/autoqueue/pending/7_ffflat_dslmast_33f_007.sch

• 170305 22:42:42

  • cyclefridge enable=2, on all rx

• 170305 15:00:00 - 170305 22:42:42

  • /home/keck/sch/mce_autotune.sch
  • /data/keckdaq/autoqueue/pending/7_ffflat_dslmast_33e_007.sch

• 170305 09:12:02

  • cyclefridge enable=2, on all rx

• 170305 01:28:58 - 170305 09:12:02

  • /home/keck/sch/mce_autotune.sch
  • /data/keckdaq/autoqueue/pending/7_ffflat_dslmast_33d_007.sch

• 170304 19:43:03

  • cyclefridge enable=2, on all rx

• 170304 12:00:54 - 170304 19:43:03

  • /home/keck/sch/mce_autotune.sch
  • /data/keckdaq/autoqueue/pending/7_ffflat_dslmast_33c_007.sch

• 170304 06:45:38

  • cyclefridge enable=2, on all rx

• 170303 23:01:29 - 170304 06:45:38

  • /home/keck/sch/mce_autotune.sch
  • /data/keckdaq/autoqueue/pending/7_ffflat_dslmast_33b_007.sch

• 170303 13:05:12

  • cyclefridge enable=2, on all rx

• 170303 05:22:41 - 170303 13:05:12

  • /home/keck/sch/mce_autotune.sch
  • /data/keckdaq/autoqueue/pending/7_ffflat_dslmast_33a_007.sch

• 170303 00:12:11

  • cyclefridge enable=2, on all rx

• 170302 16:30:00 - 170303 00:12:11

  • /home/keck/sch/mce_autotune.sch
  • /data/keckdaq/autoqueue/pending/7_ffflat_dslmast_33j_006.sch

• 170302 10:30:41

  • cyclefridge enable=2, on all rx
  • recharged comp 4

• 170302 02:48:29 - 170302 10:30:41

  • /home/keck/sch/mce_autotune.sch
  • /data/keckdaq/autoqueue/pending/7_ffflat_dslmast_33i_006.sch

• 170301 21:40:37

  • cyclefridge enable=2, on all rx

• 170301 13:57:34 - 170301 21:40:37

  • /home/keck/sch/mce_autotune.sch
  • /data/keckdaq/autoqueue/pending/7_ffflat_dslmast_33h_006.sch

• 170301 07:15:08

  • cyclefridge enable=2, on all rx

• 170228 23:32:39 - 170301 07:15:08

  • /home/keck/sch/mce_autotune.sch
  • /data/keckdaq/autoqueue/pending/7_ffflat_dslmast_33g_006.sch

• 170228 17:01:47

  • cyclefridge enable=2, on all rx

• 170228 09:20:07 - 170228 17:01:46

  • /home/keck/sch/mce_autotune.sch
  • /data/keckdaq/autoqueue/pending/7_ffflat_dslmast_33f_006.sch

• 170228 02:31:06

  • cyclefridge enable=2, on all rx

• 170227 18:49:43 - 170228 02:31:06

  • /home/keck/sch/mce_autotune.sch
  • /data/keckdaq/autoqueue/pending/7_ffflat_dslmast_33e_006.sch

• 170227 13:47:48

  • cyclefridge enable=2, on all rx

• 170227 18:49:43 -

  • /home/keck/sch/mce_autotune.sch
  • /data/keckdaq/autoqueue/pending/7_ffflat_dslmast_33e_006.sch

• 170227 13:47:48

  • cyclefridge enable=2, on all rx

• 170227 06:03:10 - 170227 13:47:47

  • /home/keck/sch/mce_autotune.sch
  • /data/keckdaq/autoqueue/pending/7_ffflat_dslmast_33d_006.sch

• 170227 00:58:36

  • cyclefridge enable=2, on all rx

• 170227 00:32:13 - 170227 00:58:36

  • /home/keck/sch/mce_autotune.sch
  • /data/keckdaq/autoqueue/pending/7_ffflat_dslmast_33d_005.sch
  • 170227 00:58:36 abort_schedule
  • installed fixed fans on unit, but problem is not overheating anymore it is
  • motor "over speed" reduced the analog in from 5v to 4.75V that corresponds to 16Hz
  • new schedule number with 006 has 16Hz chopper frequency

• 170226 16:29:51

  • cyclefridge enable=2, on all rx
  • chopper stopped during fridgecycle so scheudles are good
  • rearanged fan

• 170226 08:48:02 - 170226 16:29:51

  • /home/keck/sch/mce_autotune.sch
  • /data/keckdaq/autoqueue/pending/7_ffflat_dslmast_33c_005.sch

• 170226 00:42:21

  • cyclefridge enable=2, on all rx

• 170225 17:00:00 - 170226 00:42:21

  • /home/keck/sch/mce_autotune.sch
  • /data/keckdaq/autoqueue/pending/7_ffflat_dslmast_33b_005.sch

• 170225 10:15:39

  • cyclefridge enable=2, on all rx

• 170225 02:33:43 - 170225 10:15:38

  • /home/keck/sch/mce_autotune.sch
  • /data/keckdaq/autoqueue/pending/7_ffflat_dslmast_33a_005.sch

• 170224 21:28:17

  • cyclefridge enable=2, on all rx

  • back mirror position

  • Front top -> 57.5in (E&F)
  • Front bottom -> 23.5in (A&B)
  • Back -> 25.2in (G&H)
  • all new schedules with this mirror position will be starting with *005.sch

• 170224 13:44:25 - 170224 21:28:17

  • /home/keck/sch/mce_autotune.sch
  • /data/keckdaq/autoqueue/pending/7_ffflat_dslmast_33b_004.sch

• 170224 04:07:26

  • cyclefridge enable=2, on all rx

• 170223 20:26:12 - 170224 04:07:26

  • /home/keck/sch/mce_autotune.sch
  • /data/keckdaq/autoqueue/pending/7_ffflat_dslmast_33h_003.sch

• 170223 07:46:03

  • cyclefridge enable=2, on all rx

• 170223 03:13:08 - 170223 07:46:03

  • /home/keck/sch/mce_autotune.sch
  • /data/keckdaq/autoqueue/pending/7_ffflat_dslmast_33h_003.sch
  • 170223 07:46:03 abort_schedule
  • chopper stopped again

• 170222 04:33:23

  • cyclefridge enable=2, on all rx

• 170222 00:53:34

  • /home/keck/sch/mce_autotune.sch
  • /data/keckdaq/autoqueue/pending/7_ffflat_dslmast_33h_003.sch
  • 170222 04:33:23 abort_schedule
  • chopper stopped again
  • will re-run 33h_003.sch

• 170221 07:14:29

  • cyclefridge enable=2, on all rx

• 170220 22:37:12 - 170221 06:19:52

  • /home/keck/sch/mce_autotune.sch
  • /data/keckdaq/autoqueue/pending/7_ffflat_dslmast_33h_002.sch
  • chopper stopped again towards the end of the run, most likely not pigtails but motor controller

• 170220 17:16:35

  • cyclefridge enable=2, on all rx

• 170220 09:31:20 - 170220 17:16:35

  • /home/keck/sch/mce_autotune.sch
  • /data/keckdaq/autoqueue/pending/7_ffflat_dslmast_33j_002.sch

• 170220 04:20:18

  • cyclefridge enable=2, on all rx

• 170219 22:52:42 - 170220 04:20:18

  • /home/keck/sch/mce_autotune.sch
  • /data/keckdaq/autoqueue/pending/7_ffflat_dslmast_33j_002.sch
  • DSL chopper stopped working
  • 170220 04:20:18 abort_schedule
  • bad connection on the pigtail
  • chopper running again
  • will rerun 33j_002

• 170219 17:17:25

  • cyclefridge enable=2, on all rx

• 170219 09:32:10 - 170219 17:17:24

  • /home/keck/sch/mce_autotune.sch
  • /data/keckdaq/autoqueue/pending/7_ffflat_dslmast_33i_002.sch

• 170219 04:21:59

  • cyclefridge enable=2, on all rx

• 170218 20:40:36 - 170219 04:21:59

  • /home/keck/sch/mce_autotune.sch
  • /data/keckdaq/autoqueue/pending/7_ffflat_dslmast_33g_002.sch

• 170218 15:18:12

  • cyclefridge enable=2, on all rx

• 170218 07:34:47 - 170218 15:18:11

  • /home/keck/sch/mce_autotune.sch
  • /data/keckdaq/autoqueue/pending/7_ffflat_dslmast_33f_002.sch

• 170217 20:00:43

  • cyclefridge enable=2, on all rx

• 170217 11:13:15 - 170217 20:00:43

  • /home/keck/sch/mce_autotune.sch
  • /data/keckdaq/autoqueue/pending/7_ffflat_dslmast_33d_003.sch

• 170217 06:06:24

  • cyclefridge enable=2, on all rx

• 170216 22:24:58 - 170217 06:06:24

  • /home/keck/sch/mce_autotune.sch
  • /data/keckdaq/autoqueue/pending/7_ffflat_dslmast_33j_001.sch

• 170216 17:24:03

  • cyclefridge enable=2, on all rx

• 170216 09:41:20 - 170216 17:24:03

  • /home/keck/sch/mce_autotune.sch
  • /data/keckdaq/autoqueue/pending/7_ffflat_dslmast_33i_001.sch

• 170216 04:27:23

  • cyclefridge enable=2, on all rx

• 170215 20:44:02 - 170216 04:27:23

  • /home/keck/sch/mce_autotune.sch
  • /data/keckdaq/autoqueue/pending/7_ffflat_dslmast_33h_001.sch

• 170215 13:04:33

  • cyclefridge enable=2, on all rx

• 170215 05:23:12 - 170215 13:04:33

  • /home/keck/sch/mce_autotune.sch
  • /data/keckdaq/autoqueue/pending/7_ffflat_dslmast_33g_001.sch

• 170215 00:12:07

  • cyclefridge enable=2, on all rx

---

Winter 2017

---

• 170214 16:30:00 - 170215 00:12:07

  • /home/keck/sch/mce_autotune.sch
  • /data/keckdaq/autoqueue/pending/7_ffflat_dslmast_33f_001.sch

• 170214 10:18:20

  • cyclefridge enable=2, on all rx

• 170214 02:35:36 - 170214 10:18:20

  • /home/keck/sch/mce_autotune.sch
  • /data/keckdaq/autoqueue/pending/7_ffflat_dslmast_33c_003.sch

• 170213 17:01:59

  • cyclefridge enable=2, on all rx

• 170213 08:01:47 - 170213 17:01:59

  • /home/keck/sch/mce_autotune.sch
  • /data/keckdaq/autoqueue/pending/7_ffflat_dslmast_33b_003.sch
  • masd0 did not start
  • keck33 unresponsive, reboot, took long time for disk check
  • 170213 08:06:58 Exiting schedule: /data/keckdaq/autoqueue/pending/7_ffflat_dslmast_33b_003.sch
  • 170213 09:21:00 Starting schedule: /home/keck/sch/mce_autotune.sch
  • 170213 09:22:15 Starting schedule: /data/keckdaq/autoqueue/pending/7_ffflat_dslmast_33b_003.sch

• 170213 02:48:21

  • cyclefridge enable=2, on all rx
  • compressor 3 was quite low
  • one of omega0 PSU died (the newest one in the middle slot!?)

• 170212 19:08:05 - 170213 02:48:20

  • /home/keck/sch/mce_autotune.sch
  • /data/keckdaq/autoqueue/pending/7_ffflat_dslmast_33a_003.sch

• 170212 12:12:20

  • cyclefridge enable=2, on all rx

• 170212 04:30:03 - 170212 12:12:20

  • /home/keck/sch/mce_autotune.sch
  • /data/keckdaq/autoqueue/pending/7_ffflat_dslmast_33e_002.sch

• 170211 23:30:00

  • cyclefridge enable=2, on all rx

• 170211 15:30:00

  • /home/keck/sch/mce_autotune.sch
  • /data/keckdaq/autoqueue/pending/7_ffflat_dslmast_33e_002.sch
  • all new ffflat will start with 002 now
  • keck32 /data run full since it was still on cmb mode and tmp2 dir for purge_data.pl was deleted on omega0
  • gonna rerun7_ffflat_dslmast_33e_002.sch
  • /home/keck/sch/runend.sch
  • weather:
  • windows and baffles:

• • cyclefridge enable=2, on all rx

• 1700210

  • /home/keck/sch/mce_autotune.sch
  • 170210 11:15:41 7_ffflat_dslmast_33a_001
  • First started this schedule at 10:42:54, but aborted to run mce_autotune.sch
  • Chopper still at ~16 Hz
  • Weather cleared by the start of 33a_001.
  • /home/keck/sch/mce_autotune.sch
  • 170211 00:23:18 7_ffflat_dslmast_33d_001

• 1700209

  • /home/keck/sch/mce_autotune.sch
  • 7_ffflat_dslmast_33c_000.sch
  • 170209 05:21:40 - 170209 13:02:37
  • 16Hz chop ran all the way through? - data may be good
  • /home/keck/sch/mce_autotune.sch
  • 170209 21:05:14 - 170210 04:49:2 7_ffflat_dslmast_33e_001
  • First started this schedule at 20:56:56, but aborted after 11 seconds to run mce_autotune.sch
  • Weather is bad. Low visibility, blowing snow, 18+ knot winds

• 1700208

  • 7_ffflat_dslmast_33b_000.sch
  • 170208 00:34:00 - 170208 08:14:24
  • 12Hz chop ran all the way through - data is probably good

• 1700207

  • 7_ffflat_dslmast_33b_000.sch
  • 170207 12:06:01 - 170207 19:46:30
  • 16.6Hz chop
  • fridge ran out - and chopper wasn't running

• 1700207

  • 7_ffflat_dslmast_33a_000.sch
  • 170207 04:24:17 - 170207 12:06:01
  • 16.6Hz chop
  • chopper stopped 30 mins in with over speed error

• 1700206

  • Reinstalled rx4 membrane and adjusted flow rates on all.
  • Set minimal flow on all - up top (after time to stabilize) all membranes appear at least minimally pressurized.
  • However only two register close to recommended value on feedback gauges (D&C) - rest close to or at zero.
  • Leave it to Robert to determine if membrane convexity is adequate/normal.

• 1700205

  • 170205 21:00:00 Hoisted mirror to "new front position"
  • A,B @ 10.74in
  • E,F @ 53.84in
  • G,H @ 30.47in
  • Concerned this may push the current even higher at max el.

• 1700204

  • 170204 04:00:00 went to horizon to install FFF
  • 170204 04:30:00 raised FFF to 45deg elevation
  • 170204 04:30:00 rx0 failed to re-cool like the rest - 4K temps eventually came back down around 170205 00:00:00 - would have pumped on it but vacuum gauge was not working.

• 1700203

  • 170202 22:08:45 started keckycal_001.sch on Rx0, run 5. Weather: some clouds on horizon, -34 C. Finished 170202 23:00:12
  • 170202 23:05:57 started keckycal_001.sch on Rx0, run 6. Weather: some clouds on horizon, -34 C. Finished 170202 23:36:13
  • 170203 02:15:32 Started test procedure for determing DSC rotation axis on Rx0 for a second time, this time with the 0/45 degree measuring bar. Finished 170203 03:38:41.
  • 170203 08:07:46 started 1_keckycal_001.sch on Rx1, run 3. Weather: some clouds on horizon, -34 C. Finished 170203 08:54:34
  • 170203 08:55:08 started 1_keckycal_001.sch on Rx1, run 4. Weather: some clouds within 20 deg of zenith, -34 C. Finished 170203 09:25:25

• 1700202

  • 170201 23:56:18 Started keckycal_001.sch on Rx4, run 1. Weather is clear -33 C. Finished 170202 00:40:47.
  • 170202 00:41:57 Started keckycal_001.sch on Rx4, run 2. Weather is clear -33 C. Finished 170202 01:14:26.
  • 170202 01:14:43 Started keckycal_001.sch on Rx4, run 3. Weather is clear -33 C. Finished 170202 01:45:46. Took trailing angle measurements before the schedule was ready. It should still be ok because the DSC didn't rotate, but maybe the dk did.
  • 170202 01:46:47 Started keckycal_001.sch on Rx4, run 4. Rotated the DSC clockwise instead of the usual CCW. Finished 170202 02:31:25
  • 170202 03:56:56 Started test procedure for determing DSC rotation axis on Rx4. Finished 170202 05:08:21.
  • 170202 07:20:28 Started 9_kCMB_50_132.sch

• 1700201

  • 170201 01:25:47 Starting schedule: 1_forebaffle_on_off_4_010.sch, on all receivers
  • 170201 06:48:22 /home/keck/sch/starpoint_pole_keck_summer.sch
  • 170201 08:03:56 /home/keck/sch/starpoint_pole_keck_summer.sch
  • Both above runs pretty successful - together = good data set

• 170131

  • 170131 02:33:44 Started keckycal_001.sch on Rx3, run 1. Weather is pretty bad -32 C. Finished 170131 03:24:18.
  • 170131 03:43:29 Started keckycal_001.sch on Rx3, run 2. Hazy. Finished 170131 04:11:48.
  • 170131 04:16:23 Started keckycal_001.sch on Rx3, run 3. Hazy. Finished 170131 04:45:27.
  • 170131 07:15:16 Started test procedure for determining DSC rotation axis on Rx3. Finished 170131 08:21:18.

• 170130

  • Weather: thin clouds and blowing now.
  • 170129 21:38:45 Started keckycal_001.sch on Rx0, run 3
  • Note: we rotated the calibrator before the appropriate spot in the schedule. Then we rotated it in the opposite direction
  • 170129 22:24:35 Started keckycal_001.sch on Rx0, run 4. Finished 170129 23:14:42
  • 170130 02:50:09 Started keckycal_001.sch on Rx1, run 1. It's cloudy for this run.
  • 170130 03:52:48 Started keckycal_001.sch on Rx1, run 2. Still clouds in the sky. Finished 170130 04:33:31
  • 170130 04:40:23 Started test procedure for determining DSC rotation axis on Rx1. Finished 170130 06:14:09
  • 170130 08:36:17 Started keckycal_001.sch on Rx2, run 1. Finished 170130 09:21:23
  • 170130 09:22:02 Started keckycal_001.sch on Rx2, run 2. Finished 170130 09:57:59. Weather some clouds above. -33 C
  • 170130 10:01:21 Started keckycal_001.sch on Rx2, run 3. Finished 170130 10:41:43. Kind of a flat sky; not completely overcast, not blue.
  • 170130 21:14:37 Started test procedure for determing DSC rotation axis on Rx2. Finished 170130 22:18:48.
  • 170130 22:49:55 Started keckycal_001.sch on Rx2, run 4 (Aborted at 170130 23:17:50). Cloudy and snow is falling on membrane. One extra splash of frozen snot on highest edge (when at Zenith) of sheet.

• 170129

  • Weather slightly hazy from blowing snow. Very windy with blue overhead.
  • 170129 06:52:30 Started keckycal_001.sch on Rx0, run 1
  • Note: Membrane is not pressurized. Also, forgot to hit resume_schedule before starting to rotate, so there are an extra 4 rotations (with combine on) between the marked encoder count and where the rotation data are marked with a feature bit.
  • 170129 07:54:37 Started keckycal_001.sch on Rx0, run 2
  • Note: Using larger measuring bar for this run.

• 170128

  • Ran a test procedure for the Keckycal on Rx0 to see if we can determine the axis of rotation vs the dk rotation axis. Started 01:53:33. Finished 05:00:24

• 170125

  • Completed FTS on rx2 around 16:00 UTC. Will remove FTS tomorrow.

• 170124

  • Started FTS on rx2 around 09:00 UTC. Will use same data taking strategy as on rx0, and will continue using tarp.

• 170123

  • Completed FTS on rx0 around 15:00 UTC. Will move to rx2 tomorrow.

• 170122

  • FTS on rx0 still ongoing. Finished rot=45, then started rot=90. Noticed motors started malfuncitoning in the cold, so placed large tarp over entire FTS. This solved the problem. All subsequent FTS data this season will be taken with a tarp over the FTS.

• 170121

  • Started FTS on rx0. Strategy: take data at FTS rotation angle rot=45 deg (covers A and B pol) at all nine pointings on all tiles. Then take data at rot=0,90 (covers only A or only B) and take five pointings per tile.

• 170117

  • Rx1 compressor leaks ~10 PSI per day. Recharged to 290 PSI.
  • Took G data on Rx1 every 10mK from 270 to 400 and 425
  • Took G data on Rx3 every 10mK from 270 to 390mK

• 170115

  • Rx1 almost finished cooling back down. Coldplate temp still elevated at ~4.2K. High side compressor pressure was at 260 PSI. Charged it to Robert-recommended 285 PSI at ~01:25:00. This cooled coldplate by 0.1 K, but it's now rebounding some.

• 170112

  • SNK: Made G Measurement in rx0. Tried making G measurements in rx4, observed squid jumps in dark pixels. Retuning squids for rx4.
  • Disconnected He lines from compressor for Rx1.
  • Performed 2 pump + purge cycles on Rx1 pulse tube through mount and compressor He lines. Filled to 50 PSI between pump and purge. Then pumped on lines and PT for 4 hours and filled with He to match compressor pressure. Reconnected lines to compressor and turned on at ~ 01:45 UTC. 4K coldhead had warmed to slightly over 100K.

• 170112

  • 1701112: Second time we fridge cycled all Rx after the re-loading of Rx4-K5-270GHz on Tuesday Jan 10th. All Rx fridge cycled properly except for Rx1 which failed twice getting stuck on step 5. Step 5 is the point where the pump heaters are turned off and the cycle waits for the He4 condensation point to drop below T=4.4K.
  • Refilled Rx1 from Phi= 265PSI to Phi=285PSI.
  • Sadly that seems to have made the 4K temperatures rise, not go down
  • Decide to let Rx1-K2 warm up tp 70K and pump on it to see if it's caused by residual gas in cryostat.

• 170103

  • Refilled compressors 2 and 4 to nominal pressures.

• 161231

  • 16121331: Fridge cycle Rx0-3 so we can check the squid tuning of all of them tomorrow.

• 161213

  • 161213 03:17:45 shutting down GCP to decrease HDD load on keck32 while RAID sync occurs. (Replacing old HDDs on all keckdaq machines with new.)

  ---

  Summer 2016-17

  ---

• 161206

  • With new FP on the way, have ceased observations for 2016.
  • Congratulations on a productive 2016 observing season!

• 161205 02:53:14 -

  • /home/keck/sch/mce_autotune.sch
  • /data/keckdaq/autoqueue/pending/9_kCMB_50_132.sch
  • 161205 03:35:12 Az fatal following error
  • 161205 05:46:48 abort schedule
  • Debugging az fatal following error - hear nothing with stethoscope, but notice some variability in mount motion rate as driven with the hand-controller box.
  • Sit near edge of upper mount as mount being driven in azimuth - opinion is that “jerking” is worse than even a couple of weeks ago.
  • Stopping observations (and locked out mount) until discussion with others occurs.
  • /home/keck/sch/runend.sch
  • weather: partly cloudy,
  • windows and baffles:

• 161204 20:53:07

  • cyclefridge enable=2, on all rx
  • recharged compressor 2 (rx1)

• 161203 02:45:50 - 161204 20:50:30

  • /home/keck/sch/mce_autotune.sch
  • /data/keckdaq/autoqueue/pending/9_kCMB_50_131.sch
  • /home/keck/sch/runend.sch
  • weather: high clouds, clearing, partly cloudy
  • windows and baffles: clear

• 161202 21:00:57

  • cyclefridge enable=2, on all rx

• 161201 03:09:20 - 161202 20:58:21

  • /home/keck/sch/mce_autotune.sch
  • /data/keckdaq/autoqueue/pending/9_kCMB_50_130.sch
  • /home/keck/sch/runend.sch
  • weather: partly cloudy, clearing, blowing snow
  • windows and baffles: some snow/ice in baffles

• 161130 21:08:50

  • cyclefridge enable=2, on all rx

• 161129 02:55:02 - 161130 21:06:13

  • /home/keck/sch/mce_autotune.sch
  • /data/keckdaq/autoqueue/pending/9_kCMB_50_129.sch
  • /home/keck/sch/runend.sch
  • weather: partly cloudy
  • windows and baffles: clear

• 161128 21:16:42

  • cyclefridge enable=2, on all rx

• 161127 03:00:44 - 161128 21:14:05

  • /home/keck/sch/mce_autotune.sch
  • /data/keckdaq/autoqueue/pending/9_kCMB_50_128.sch
  • 161127 21:00:00 Started signalhound calibration with active use of radios and signal source.
  • 161127 21:45:00 Done with calibration.
  • /home/keck/sch/runend.sch
  • weather: partly cloudy
  • windows and baffles: clear

• 161126 21:24:34

  • cyclefridge enable=2, on all rx

• 161125 03:40:40 - 161126 21:21:57

  • /home/keck/sch/mce_autotune.sch
  • /data/keckdaq/autoqueue/pending/9_kCMB_50_127.sch
  • /home/keck/sch/runend.sch
  • weather: partly cloudy
  • windows and baffles: clear

• 161124 21:32:25

  • cyclefridge enable=2, on all rx

• 161123 03:26:33 - 161124 21:29:49

  • /home/keck/sch/mce_autotune.sch
  • /data/keckdaq/autoqueue/pending/9_kCMB_50_126.sch
  • 161123 20:50 Recharged comp 3 (rx0) from 260psi to 275psi. Leak checking az lines while scanning didn't find anything. Will check again during fridge cycle when mount can be moved manually.
  • /home/keck/sch/runend.sch
  • weather: partly cloudy, blowing snow, clearing
  • windows and baffles: snow and ice along base

• 161122 21:40:18

  • cyclefridge enable=2, on all rx
  • Installed new forebaffle variacs

• 161121 04:04:22 - 161122 21:37:41

  • /home/keck/sch/mce_autotune.sch
  • /data/keckdaq/autoqueue/pending/9_kCMB_50_125.sch
  • 161121 04:08:23 MCE4 bolometers not showing any data. Restarted masd4 and restarted schedule.
  • Still not responding. Rebooting keck37. Got stuck with a disk check... Missed getting connected again before B01.
  • MCE4 didn't return on B02. I don't know how to get it reconfigured except by redoing CMB startup sequence, so abort schedule.
  • 161121 05:28:16 abort_schedule
  • 161121 05:28:37 redo mce_autotune.sch
  • 161121 05:30:02 restart CMB schedule
  • MCE4 still not taking data, and then all updates to GCP pages stopped, so restart all masd and GCP. Then MCE4 dsp won't respond, so try cold rebooting keck37 again.
  • Even after reboot and restart of schedules, still not getting data from MCE4 - appears to be problem in initializing masd4. Sent email to keckarray list.
  • Will run from B03 onward with just rx0-3 until advice from others received.
  • 161121 20:35:02 suspend_schedule - deal with az-diff jump
  • Power-cycled MCE4. mce_status returns with no errors, so going to try restarting the CMB schedule to see if rx4 data will now be taken.
  • 161121 20:49:39 Exiting schedule
  • Power-cycled MCE4. mce_status returns with no errors, so going to try restarting the CMB schedule to see if rx4 data will now be taken.
  • Still not acquiring data
  • Did nothing new, but mce4 came back to life for D01.
  • /home/keck/sch/runend.sch
  • weather: clear, partly cloudy
  • windows and baffles: clear

• 161120 21:49:33

  warm
  • Swapped compressor 4 (after topping up oil level) in for compressor 5.
  • Topped up compressor 4 helium to get hi-side pressure to ~295psi. Smaller than normal pressure differential, with low-side pressure ~130psi.
  • 161108 00:02:43 Station power outage. Temporary stop in scans during D02.
  • 161108 00:08:13 Took manual control of mount to oil the az wrap which started to whine.
  • 161108 00:14:02 Returned PMAC control to GCP
  • 161108 00:29:00 Turn off compressor for rx4 (compressor 4)
  • Swap compressor 5 back in to running rx4; compressor 4 back into hallway as a spare.
  • 161108 00:37:00 rx4 compressor turned back on
  • 161108 02:33:00 rx4 compressor pressure a bit high now (extra He from raising pressure when on compressor 4??). Released He until hi-side ~195psi.
  • 161108 18:47:38 MCE1 stopped taking data. Restarted masd1.
  • Did not come back with next scanset. Reboot keck34.
  • 161108 20:37:35 MCE1 still not taking data. Did shutdown and cold startup of keck34.
  • /home/keck/sch/runend.sch
  • weather: cloudy, partly cloudy
  • windows and baffles: snow and ice in baffles and around edges of windows

• 161106 22:43:12

  • cycleFridge enable=2, on all rx

• 161105 04:34:54 - 161106 22:40:35

  • /home/keck/sch/mce_autotune.sch
  • /data/keckdaq/autoqueue/pending/9_kCMB_50_117.sch
  • 161105 05:15:57 - mistake by summer-over-in-training during az track cleaning caused az-diff jump; fixed between sky-dip and B01 scans
  • recharged compressor for rx2
  • /home/keck/sch/runend.sch
  • weather: overcast, blowing snow, clearing at ground level, overcast
  • windows and baffles: minor amount of snow in and on baffles

• 161104 22:51:05

  • cycleFridge enable=2, on all rx

• 161103 04:25:09 - 161104 22:51:05

  • /home/keck/sch/mce_autotune.sch
  • /data/keckdaq/autoqueue/pending/9_kCMB_50_116.sch
  • 161103 12:18 antenna/mediator crashed
  • restarted masd gcp
  • 161103 12:25:02 schedule /home/keck/sch/mce_autotune.sch
  • 161103 12:26:50 Starting schedule: /data/keckdaq/autoqueue/pending/9_kCMB_50_116.sch
  • /home/keck/sch/runend.sch
  • weather: thin clouds, clearing, overcast, blowing snow
  • windows and baffles: windows clear, little snow in and on the baffle

• 161102 22:58:57

  • cycleFridge enable=2, on all rx
  • swapped compressor 4 with spare, bad vibration
  • according to cyromech data, looks like at least 3 out 6 compressors need addtional oil !!!!

• 161101 04:29:32 - 161102 22:58:57

  • /home/keck/sch/mce_autotune.sch
  • /data/keckdaq/autoqueue/pending/9_kCMB_50_115.sch
  • comp4 still losing pressure, haven't found the leak yet
  • /home/keck/sch/runend.sch
  • weather: clear, haze, thin clouds
  • windows and baffles: clear

• 161031 23:06:48

  • cycleFridge enable=2, on all rx
  • D high had a massive wear through the outer breading behind the ladder, but also D low had a small leak
  • both lines were installed Nov 2015, installed 2 new lines
  • comp 4 has some vibration, probably should get a new motor unit for that one as well
  • rx2 is still good and going through normal fridgecycle

• 161030 04:51:47 - 161031 23:06:48

  • /home/keck/sch/mce_autotune.sch
  • /data/keckdaq/autoqueue/pending/9_kCMB_50_114.sch
  • 161031 06:15 rx3 warming up, massiv He leak, can't tell which line yet, since it
  • seems right behind the ladder
  • restarted masd3
  • rx3 recording data again
  • at the momemt it looks like we got 2 leaks one in D high, one in D low
  • /home/keck/sch/runend.sch
  • weather: thin clouds, haze, overcast, clearing
  • windows and baffles: clear

• 161029 23:14:41

  • cycleFridge enable=2, on all rx
  • no pressure loss on rx2, still going strong

• 161028 06:15:59 - 161029 23:14:41

  • /home/keck/sch/mce_autotune.sch
  • rx2 didn't make it also the second fridgecycle looked promesing
  • /data/keckdaq/autoqueue/pending/9_kCMB_50_113.sch
  • 161028 07:55:34 abort_schedule
  • starting run with rx2 as well
  • recharged comp 1 even higher to 315psi, 4K is now back in the normal range
  • 161028 07:55:40 schedule /home/keck/sch/mce_autotune.sch
  • 161028 07:58:10 Starting schedule: /data/keckdaq/autoqueue/pending/9_kCMB_50_113.sch
  • rx2 staying cold, now the lowest 4K
  • /home/keck/sch/runend.sch
  • weather: thin clouds, mostly clear, clear, clouds moving in
  • windows and baffles:

• 161027 23:22:40

  • cycleFridge enable=2, on all rx
  • recharged comp1,4
  • we might have another slight leak on comp1, which might explain the strange behavior on rx2
  • charging it up to 300psi, seems to help with the 4K temperature, pressure on compressor changes quite a bit
  • during different times of the run

• 161026 05:01:05 - 161027 23:22:40

  • /home/keck/sch/mce_autotune.sch
  • /data/keckdaq/autoqueue/pending/9_kCMB_50_112.sch
  • the 4k warmup on rx2 through it off again, running that run with only 4rx again
  • since rx2 cooled down well and looks good, I restarted the cmb schedule just before the gal scan sets
  • 161026 23:28:05 abort_schedule
  • 161026 23:28:38 schedule /home/keck/sch/mce_autotune.sch
  • 161026 23:29:54 Starting schedule: /data/keckdaq/autoqueue/pending/9_kCMB_50_112.sch
  • ok rx2 didn't make it all the way, started to warm up slightly over 300mK during the last scansets
  • /home/keck/sch/runend2.sch
  • weather: clear overhead, haze, blowing snow
  • windows and baffles: windows clear, very little snow on the forebaffles

• 161025 23:41:32

  • cycleFridge enable=2, on all rx
  • started rx2 cycle one scanset early
  • 161025 22:37:42 cycleFridge enable=2, rx=rx2
  • well, that didn't work, even for the other cycles, rx2 was disabled
  • something happened and rx2 4K warmed up to 10K
  • el current drifting apart

• 161024 06:10:30 - 161025 23:30:35

  • missed first scanset, but running on all 5rx
  • /home/keck/sch/mce_autotune.sch
  • /data/keckdaq/autoqueue/pending/9_kCMB_50_111.sch
  • /home/keck/sch/runend2.sch
  • weather: mostly clear overhead, blowing snow
  • windows and baffles: windows clear, very little snow on the baffles

• 161023 23:39:38

  • cycleFridge enable=2, on all rx
  • comp 4 has elevated noise
  • rx2 cycle still the slowest, but might just make the first scanset
  • checked osborne rollers, are tight in some places and not touching on others, did not adjust them since last adjustment

• 161022 05:43:54 - 161023 23:39:38

  • /home/keck/sch/mce_autotune.sch
  • /data/keckdaq/autoqueue/pending/9_kCMB_50_110.sch
  • running without rx2
  • 161022 20:50:10 fridgecycle on rx2
  • did a normal cycle, 4K temperatures still a bit elevated
  • /home/keck/sch/runend2.sch
  • weather: clear, haze, blowing snow
  • windows and baffles: windows clear, very little snow on the baffles

• 161022 00:18:12

  • cycleFridge enable=2, on rx1,2,3,4
  • recharged comp 1,2

• 161020 05:23:01 - 161022 00:18:12

  • /home/keck/sch/mce_autotune.sch
  • /data/keckdaq/autoqueue/pending/9_kCMB_50_109.sch
  • running without rx2
  • 161020 10:01 unplanned power outage, for about 5min no significant warmup
  • /home/keck/sch/runend2.sch
  • weather: hazy, blowing snow
  • windows and baffles:

• 161020 00:03:06

  • cycleFridge enable=2, on all rx

• 161018 10:20:46 - 161019 23:54:09

  • missed first scansets because of rx2 slow cycle
  • /home/keck/sch/mce_autotune.sch
  • /data/keckdaq/autoqueue/pending/9_kCMB_50_108.sch
  • mediator crashed, no alarm
  • restarted gcp, masd
  • 161019 03:06:42 schedule /home/keck/sch/mce_autotune.sch
  • 161019 03:08:46 Starting schedule: /data/keckdaq/autoqueue/pending/9_kCMB_50_108.sch
  • rx2 warming up just during last scansets
  • /home/keck/sch/runend2.sch
  • 161020 00:01:00 planned power outage for generator swap
  • weather: mostly clear, blowing snow, haze
  • windows and baffles:

• 161018 00:01:52

  • cycleFridge enable=2, on all rx
  • rx2 cycle will take much longer again, FPU goes over 4K, 4K coldhead seems a bit warmer

• 161016 07:18:43 - 161018 00:01:52

  • /home/keck/sch/mce_autotune.sch
  • /data/keckdaq/autoqueue/pending/9_kCMB_50_107.sch
  • rx2 coldhead about 0.5K warmer than the others, contamination!??
  • /home/keck/sch/runend.sch
  • weather: haze, mostly clear
  • windows and baffles: clear

• 161016 00:09:43

  • cycleFridge enable=2, on all rx
  • cycle on rx2 unusally long, but pressure is good
  • otherwise cycle doesn't look abnormal besides being longer?
  • not gonna make the first scanset, rx2 still cooling down

• 161014 05:36:27 - 161016 00:09:43

  • /home/keck/sch/mce_autotune.sch
  • /data/keckdaq/autoqueue/pending/9_kCMB_50_106.sch
  • /home/keck/sch/runend.sch
  • weather: mostly clear, haze, some snow, thin clouds
  • windows and baffles: windows clear, some snow in and on the baffles

• 161014 00:17:34

  • cycleFridge enable=2, on all rx
  • He leak in D Hi, swapped for spare line
  • spduws1 froze, restarted gcp, masd

• 161012 05:54:15 - 161014 00:17:34

  • /home/keck/sch/mce_autotune.sch
  • /data/keckdaq/autoqueue/pending/9_kCMB_50_105.sch
  • az-diff jump, had to readjust readhead 3A
  • data are good after 161012 11:54:04 resume_schedule
  • 161012 18:19:47 az fatal following error
  • 161012 around 23:15 more fatal following errors
  • looks like another busted gear box
  • yes more broken gears !!!
  • 161013 01:23:10 resume_schedule
  • /home/keck/sch/runend.sch
  • weather: mostly clear, some haze
  • windows and baffles: clear

• 161012 00:25:03

  • cycleFridge enable=2, on all rx
  • recharged comp 4 leak!??

• 161010 06:23:56 - 161012 00:25:03

  • /home/keck/sch/mce_autotune.sch
  • /data/keckdaq/autoqueue/pending/9_kCMB_50_104.sch
  • 161010 08:32:47 suspend_schedule
  • az-diff jump, replaced readhead roller 4A
  • 161010 08:55:00 resume_schedule
  • 161010 13:08:08 el current drifting apart
  • masd 0 stopped, had to reboot keck33
  • 161011 11:41:08 fatal following error :(
  • hope it is not starting again with mount problems
  • /home/keck/sch/runend.sch
  • weather: clearing, partly overcast
  • windows and baffles: clear

• 161010 00:44:48

  • cycleFridge enable=2, on all rx
  • recharged comp 1,3,4

• 161008 06:04:15 - 161010 00:33:18

  • /home/keck/sch/mce_autotune.sch
  • /data/keckdaq/autoqueue/pending/9_kCMB_50_103.sch
  • /home/keck/sch/runend.sch
  • weather: clear, overcast, blowing snow
  • windows and baffles: windows clear, quite some snow in and on baffles

• 161008 00:41:10

  • cycleFridge enable=2, on all rx

• 161006 06:13:50 - 161008 00:41:10

  • /home/keck/sch/mce_autotune.sch
  • /data/keckdaq/autoqueue/pending/9_kCMB_50_102.sch
  • /home/keck/sch/runend.sch
  • weather: mostly clear, clear
  • windows and baffles: clear

• 161006 00:49:00

  • cycleFridge enable=2, on all rx

• 161004 06:50:00 - 161006 00:49:00

  • /home/keck/sch/mce_autotune.sch
  • /data/keckdaq/autoqueue/pending/9_kCMB_50_101.sch
  • /home/keck/sch/runend.sch
  • weather: clear, getting overcast
  • windows and baffles: clear

• 161004 00:56:53

  • cycleFridge enable=2, on all rx
  • today fridgecycle on rx1 takes normal time again

• 161002 05:46:58 - 161004 00:56:53

  • /home/keck/sch/mce_autotune.sch
  • /data/keckdaq/autoqueue/pending/9_kCMB_50_100.sch
  • 161003 11:34:00 rx1 warming up
  • so there was an influence from the fast fridge cycle (see below)
  • /home/keck/sch/runend1.sch
  • weather: clear
  • windows and baffles: clear

• 161002 01:15:11

  • cycleFridge enable=2, on all rx
  • 161002 02:50:39 Starting schedule: /home/keck/sch/starpoint_pole_keck_summer.sch
  • 161002 04:37:11 Exiting schedule: /home/keck/sch/starpoint_pole_keck_summer.sch
  • some unusual fast cooldown on rx1, but everything looks normal

• 160930 06:38:33 - 161002 01:15:11

  • /home/keck/sch/mce_autotune.sch
  • /data/keckdaq/autoqueue/pending/9_kCMB_50_099.sch
  • problems with keckViewer and monitor
  • did lose a bunch of data, no alarm/pager initiated
  • /home/keck/sch/runend.sch
  • weather: clear, cold, overcast, clearing
  • windows and baffles: windows clear, litte snow on the baffles

• 160930 01:12:37

  • cycleFridge enable=2, on all rx
  • running all 5 baffle heaters on one variac on minimal heat

• 160928 06:50:22 - 160930 01:12:37

  • /home/keck/sch/mce_autotune.sch
  • /data/keckdaq/autoqueue/pending/9_kCMB_50_098.sch
  • /home/keck/sch/runend.sch
  • weather: clear
  • windows and baffles: clear

• 160928 01:20:31

  • cycleFridge enable=2, on all rx

• 160926 06:52:49 - 160928 01:20:31

  • /home/keck/sch/mce_autotune.sch
  • /data/keckdaq/autoqueue/pending/9_kCMB_50_097.sch
  • /home/keck/sch/runend.sch
  • weather: clear, mostly clear
  • windows and baffles: clear

• 160926 01:28:19

  • cycleFridge enable=2, on all rx
  • recharged comp 2

• 160924 07:29:20 - 160926 01:28:19

  • /home/keck/sch/mce_autotune.sch
  • /data/keckdaq/autoqueue/pending/9_kCMB_50_096.sch
  • /home/keck/sch/runend.sch
  • weather: high winds, blowing snow, overcast, clearing
  • windows and baffles: windows clear, little snow in the baffles

• 160924 01:36:15

  • cycleFridge enable=2, on all rx
  • el current started to drift apart
  • some problems with the baffle heat, some strange cross conection between the 2 variacs

• 160922 07:08:18 - 160924 01:36:15

  • /home/keck/sch/mce_autotune.sch
  • /data/keckdaq/autoqueue/pending/9_kCMB_50_095.sch
  • 160923 04:49:20 suspend_schedule, readhead roller B1 bearing ceased, replaced roller
  • 160923 05:11:12 resume_schedule
  • /home/keck/sch/runend.sch
  • weather: mostly clear, wind picking up, blowing snow, high winds, overcast
  • windows and baffles: windows clear, quite some snow in and on the baffles

• 160922 01:44:05

  • cycleFridge enable=2, on all rx
  • recharged comp 3

• 160920 07:45:03 - 160922 01:44:05

  • /home/keck/sch/mce_autotune.sch
  • /data/keckdaq/autoqueue/pending/9_kCMB_50_094.sch
  • /home/keck/sch/runend.sch
  • weather: mostly clear
  • windows and baffles: windows clear, little bit of snow on the baffles

• • cycleFridge enable=2, on all rx

• 160918 07:22:36 - 160920 01:51:55

  • /home/keck/sch/mce_autotune.sch
  • /data/keckdaq/autoqueue/pending/9_kCMB_50_093.sch
  • /home/keck/sch/runend.sch
  • weather: mostly clear
  • windows and baffles: windows clear, very little snow on the baffles

• 160918 01:59:22

  • cycleFridge enable=2, on all rx

• 160916 07:47:49 - 160918 01:59:22

  • /home/keck/sch/mce_autotune.sch
  • /data/keckdaq/autoqueue/pending/9_kCMB_50_092.sch
  • masd4 stopped towards the end of the run
  • rebooted keck37, restarted masd4
  • /home/keck/sch/runend.sch
  • weather: clear, blowing snow, mostly clear
  • windows and baffles: windows clear, very little snow in the baffles

• 160916 02:07:39

  • cycleFridge enable=2, on all rx
  • recharged comp 1,4
  • el motor current drifted apart
  • spudws1 froze, restarted gcp, masd

• 160914 07:37:53 - 160916 02:07:39

  • /home/keck/sch/mce_autotune.sch
  • /data/keckdaq/autoqueue/pending/9_kCMB_50_091.sch
  • /home/keck/sch/runend.sch
  • weather: overcast, blowing snow, clearing
  • windows and baffles: windows clear, some snow in the baffles

• 160914 02:15:33

  • cycleFridge enable=2, on all rx

• 160912 08:18:09 - 160914 02:15:33

  • /home/keck/sch/mce_autotune.sch
  • /data/keckdaq/autoqueue/pending/9_kCMB_50_090.sch
  • no pmac problems with fixed hand controller
  • /home/keck/sch/runend.sch
  • weather: mostly clear, wind picking up, blowing snow, overcast
  • windows and baffles: windows clear, lots of snow in and on the baffles

• 160912 03:01:29

  • cycleFridge enable=2, on all rx

• 160910 08:24:22 - 160912 02:23:23

  • /home/keck/sch/mce_autotune.sch
  • /data/keckdaq/autoqueue/pending/9_kCMB_50_089.sch
  • /home/keck/sch/runend.sch
  • installed fixed hand controller box, restarted keck31 several times
  • restarted gcp and masd
  • weather: clear, little hazy
  • windows and baffles: windows clear, some snow in the baffles

• 160910 02:31:14

  • cycleFridge enable=2, on all rx
  • reseated connectors on rx4

• 160908 08:06:41 - 160910 02:31:14

  • /home/keck/sch/mce_autotune.sch
  • /data/keckdaq/autoqueue/pending/9_kCMB_50_088.sch
  • /home/keck/sch/runend.sch
  • weather: wind dying again, clear, cold
  • windows and baffles: windows clear, some snow in and on baffles

• 160908 02:39:08

  • cycleFridge enable=2, on all rx

• 160906 08:47:35 - 160908 02:39:08

  • /home/keck/sch/mce_autotune.sch
  • /data/keckdaq/autoqueue/pending/9_kCMB_50_087.sch
  • 160907 02:44 40s planned power outage, no impact on SPUD
  • /home/keck/sch/runend.sch
  • weather: clear, very cold, wind picking up, getting hazy
  • windows and baffles: windows clear, lots of snow in the baffles

• 160906 03:28:41

  • cycleFridge enable=2, on all rx
  • hand controller doesn't work with old and new cable
  • omega0 power supply died (actually one of the new ones installed in July)

• 160904 09:20:25 - 160906 02:46:59

  • /home/keck/sch/mce_autotune.sch
  • /data/keckdaq/autoqueue/pending/9_kCMB_50_086.sch
  • /home/keck/sch/runend.sch
  • weather: clear
  • windows and baffles: clear

• 160904 03:03:18

  • cycleFridge enable=2, on all rx
  • 160904 04:59:42 Starting schedule: /home/keck/sch/starpoint_pole_keck_winter.sch
  • 160904 09:20:25 Exiting schedule: /home/keck/sch/starpoint_pole_keck_winter.sch

• 160902 08:18:46 - 160904 02:54:50

  • /home/keck/sch/mce_autotune.sch
  • /data/keckdaq/autoqueue/pending/9_kCMB_50_085.sch
  • /home/keck/sch/runend.sch
  • weather: mostly clear, clear
  • windows and baffles: clear

• 160902 03:02:42

  • cycleFridge enable=2, on all rx
  • recharged comp 1,2

• 160831 09:12:10 - 160902 03:02:42

  • /home/keck/sch/mce_autotune.sch
  • /data/keckdaq/autoqueue/pending/9_kCMB_50_084.sch
  • yeah no drive problems !! :)
  • /home/keck/sch/runend.sch
  • weather: mostly clear, some clouds
  • windows and baffles: windows clear, some snow in the baffles

• 160831 04:03:03

  • cycleFridge enable=2, on all rx

• 160830 02:41:07

  • cycleFridge enable=2, on all rx
  • pmac is behaving stranger and stranger
  • replaced one Osborne roller (3A)
  • pmac unresponsive, telescope locked out
  • quite a few times the el motor current driffted apart as well
  • 160931 removed keck31, took hand controller off
  • pmac is running now, not sure yet what the cause was, but
  • testing now, also during antenna/gcp restart, rx4 went into
  • an undefined state and started warming up

• 160828 08:46:39 - 160830 02:41:07

  • /home/keck/sch/mce_autotune.sch
  • /data/keckdaq/autoqueue/pending/9_kCMB_50_083.sch
  • 160828 09:52 pmac stopped, luckely before the runstart
  • 160828 10:15 pmac stopped
  • 160828 10:20 pmac stopped
  • 160830 01:01 pmac stopped
  • 160830 01:28 pmac stopped
  • 160830 01:30 pmac stopped frequently
  • 160830 02:38:28 abort_schedule
  • /home/keck/sch/runend.sch
  • weather: hazy
  • windows and baffles:

• 160828 03:22:23

  • cycleFridge enable=2, on all rx

• 160827 06:09:41 - 160828 03:22:23

  • /home/keck/sch/mce_autotune.sch
  • /data/keckdaq/autoqueue/pending/9_kCMB_50_082.sch
  • pmac stopped (NOT fol error, didn't trip just stopped)
  • 160827 08:12>pmac stopped (NOT fol error, didn't trip just stopped)
  • and another time
  • /home/keck/sch/runend.sch
  • weather: clear, very hazy
  • windows and baffles: windows clear, some snow on the baffles

• 160826 04:05

  • az movment not really possible any more
  • most likely bad bearing on axis 2
  • telescope will be unable to move in az for at least several days

• 160826 03:07:06

  • cycleFridge enable=2, on all rx

• 160824 09:08:32 - 160826 03:07:06

  • /home/keck/sch/mce_autotune.sch
  • /data/keckdaq/autoqueue/pending/9_kCMB_50_081.sch
  • pmac stopped several times
  • now AGAIN pmac fatal fol error !!????
  • telescopes moves fine with hand controller, nothing seems wrong
  • el motot current starting to drift apart several times
  • shimed key and keyway on az drive
  • fatal following errors increasing
  • swapped az motor for spare
  • 160826 03:04:28 abort_schedule
  • /home/keck/sch/runend.sch
  • weather: clear, clear overhead
  • windows and baffles: clear

• 160824 03:49:30

  • cycleFridge enable=2, on all rx

• 160822 09:39:05 - 160824 03:38:07

  • /home/keck/sch/mce_autotune.sch
  • /data/keckdaq/autoqueue/pending/9_kCMB_50_080.sch
  • /home/keck/sch/runend.sch
  • weather: clear
  • windows and baffles: windows clear, some snow in the baffles

• 160822 03:45:59

  • cycleFridge enable=2, on all rx
  • recharged comp 4

• 160820 09:15:52 - 160822 03:45:59

  • /home/keck/sch/mce_autotune.sch
  • /data/keckdaq/autoqueue/pending/9_kCMB_50_079.sch
  • /home/keck/sch/runend.sch
  • weather: overcast, clearing, clear, power plant exhaust in MAPO direction
  • windows and baffles: clear

• 160820 03:53:51

  • cycleFridge enable=2, on all rx
  • indeed about 0.5degrees play on the key, about 25 degrees at the motor
  • key was worn, but not as bad as last year, keyways looked ok
  • swapped gearbox for a spare, current gearbox dosen't seem to run perfektly round
  • el motot current starting to drift apart

• 160818 10:33:01 - 160820 03:53:51

  • /home/keck/sch/mce_autotune.sch
  • /data/keckdaq/autoqueue/pending/9_kCMB_50_078.sch
  • period with fatal fol err, gonna check for play of the key during cycle
  • /home/keck/sch/runend.sch
  • weather: clear,l hazy, totally overcast, warm
  • windows and baffles: windows clear, some snow on the baffles

• 160818 05:28:01

  • cycleFridge enable=2, on all rx
  • el motor current starting to drift apart

• 160818 04:01:42

  • cycleFridge enable=2, on all rx
  • el motor current starting to drift apart
  • spudws 1 froze, gcp strange state, had to restart keck31
  • had to restart fridgecycle

• 160816 09:23:18 - 160818 04:01:42

  • /home/keck/sch/mce_autotune.sch
  • /data/keckdaq/autoqueue/pending/9_kCMB_50_077.sch
  • 160816 22:11 power outage
  • 160816 22:13 power on again
  • 160817 14:00 power outage for 2 min
  • /home/keck/sch/runend.sch
  • weather: clear, high winds, blowing snow, wind dying, clear
  • windows and baffles: clear

• 160816 04:09:34

  • cycleFridge enable=2, on all rx
  • el motot current starting to drift apart

• 160815 02:49:45 - 160816 04:09:34

  • /home/keck/sch/mce_autotune.sch
  • /data/keckdaq/autoqueue/pending/9_kCMB_50_076.sch continued
  • /home/keck/sch/runend.sch
  • weather: clear overhead, blowing snow
  • windows and baffles: clear

• 160814 23:28:29

  • single cycle since UC and IC started to run out
  • cycleFridge enable=1, on all rx

• 160814 09:42:23 - 160814 22:19:05

  • /home/keck/sch/mce_autotune.sch
  • pmac stopped
  • /data/keckdaq/autoqueue/pending/9_kCMB_50_076.sch
  • 160814 21:53 power outage
  • all receivers warmed up to about 350mK
  • stopped masd and gcp, started to shut down computers
  • 160814 22:23:53 power back on
  • /home/keck/sch/runend.sch
  • weather: hazy, blowing snow, clear

• 160814 04:17:27

  • cycleFridge enable=2, on all rx
  • el motor current starting to drift apart

• 160812 10:18:24 - 160814 04:17:27

  • /home/keck/sch/mce_autotune.sch
  • /data/keckdaq/autoqueue/pending/9_kCMB_50_075.sch
  • pmac stopped, fatal fol err on az
  • /home/keck/sch/runend.sch
  • weather: overcast, warm, hazy, wind picking up, blowing snow
  • windows and baffles: windows clear, some snow in the baffles

• 160812 04:37:45

  • cycleFridge enable=2, on all rx
  • el motor current starting to drift apart

• 160810 10:26:16 - 160812 04:25:18

  • /home/keck/sch/mce_autotune.sch
  • /data/keckdaq/autoqueue/pending/9_kCMB_50_074.sch
  • pmac stopped, fatal fol err on az
  • /home/keck/sch/runend.sch
  • weather: clear overhead, high winds, blowing snow, wind dying, overcast
  • windows and baffles: windows clear, very little snow on the baffles

• 160810 04:47:11

  • cycleFridge enable=2, on all rx
  • recharged comp 1

• 160808 09:58:02 - 160810 04:33:11

  • /home/keck/sch/mce_autotune.sch
  • /data/keckdaq/autoqueue/pending/9_kCMB_50_073.sch
  • /home/keck/sch/runend.sch
  • weather: hazy, high winds, blowing snow
  • windows and baffles: windows clear, quite some snow in and on the baffles

• 160808 04:41:02

  • cycleFridge enable=2, on all rx
  • swapped bad line for new spare

• 160806 10:56:52 - 160808 04:41:02

  • /home/keck/sch/mce_autotune.sch
  • /data/keckdaq/autoqueue/pending/9_kCMB_50_072.sch
  • pmac stopped, fatal fol err on az
  • /home/keck/sch/runend.sch
  • weather: clear, getting hazy, blowing snow, high winds
  • windows and baffles: windows clear, quite some snow on all the baffles

• 160806 05:01:53

  • cycleFridge enable=2, on all rx
  • pmac stops in previous run might be due to a faulty cable to the hand controller box
  • cable is very old and should be replaced
  • recharged comp 2
  • 160806 06:33:57 Starting schedule: /home/keck/sch/starpoint_pole_keck_winter.sch
  • 160806 10:56:30 Exiting schedule: /home/keck/sch/starpoint_pole_keck_winter.sch

• 160804 10:20:01 - 160806 04:48:54

  • /home/keck/sch/mce_autotune.sch
  • /data/keckdaq/autoqueue/pending/9_kCMB_50_071.sch
  • massive He leak in rx0, A Hi same again !
  • 160806 00:20 pmac stopped, no errors ?????
  • 160806 00:20 pmac stopped again, no errors ???
  • swapped A hi to spare hose
  • /home/keck/sch/runend.sch
  • 160806 01:12 pmac stopped again, no errors
  • rx0 FPU blow 300mk again
  • weather: clear, wind picking up, blowing snow
  • windows and baffles: windows clear, little bit of snow on the baffles

• 160804 04:56:45

  • cycleFridge enable=2, on all rx

• 160802 10:28:47 - 160804 04:56:45

  • /home/keck/sch/mce_autotune.sch
  • /data/keckdaq/autoqueue/pending/9_kCMB_50_070.sch
  • /home/keck/sch/runend.sch
  • weather: hazy, blowing snow, wind dying down, clear
  • windows and baffles: clear

• 160802 05:04:35

  • cycleFridge enable=2, on all rx
  • recharged comp 3

• 160731 10:43:26 - 160802 05:04:35

  • /home/keck/sch/mce_autotune.sch
  • /data/keckdaq/autoqueue/pending/9_kCMB_50_069.sch
  • /home/keck/sch/runend.sch
  • weather: clear overhead, blowing snow
  • windows and baffles: windows clear, some snow on the baffles

• 160731 05:12:28

  • cycleFridge enable=2, on all rx

• 160729 10:42:34 - 160731 05:12:28

  • /home/keck/sch/mce_autotune.sch
  • /data/keckdaq/autoqueue/pending/9_kCMB_50_068.sch
  • /home/keck/sch/runend.sch
  • weather: clear overhead, blowing snow
  • windows and baffles: windows clear, very little snow in the baffles

• 160729 05:20:21

  • cycleFridge enable=2, on all rx

• 160727 12:27:43 - 160729 05:20:21

  • /home/keck/sch/mce_autotune.sch
  • /data/keckdaq/autoqueue/pending/9_kCMB_50_067.sch
  • /home/keck/sch/runend.sch
  • weather: hazy, blowing snow, clearing, clear overhead
  • windows and baffles: windows clear, very little snow in the baffles

• 160727 07:07:00

  • cycleFridge enable=2, on all rx

• 160727 05:28:12

  • cycleFridge enable=2, on all rx
  • swapped 5v power supply for syncbox
  • problems with antenna layer
  • had to restart keck31, took a while due to disk check
  • fridge cycle aborted
  • restarted gcp and masd several times

• 160725 11:18:36 - 160727 05:28:12

  • /home/keck/sch/mce_autotune.sch
  • /data/keckdaq/autoqueue/pending/9_kCMB_50_066.sch
  • /home/keck/sch/runend.sch
  • weather: clear, wind picking up, slight haze, blowing snow
  • windows and baffles: windows clear,

• 160725 05:35:39

  • cycleFridge enable=2, on all rx
  • recharged comp 4

• 160723 11:37:02 - 160725 05:35:39

  • /home/keck/sch/mce_autotune.sch
  • /data/keckdaq/autoqueue/pending/9_kCMB_50_065.sch
  • masd1 stopped, restarted keck34, masd1
  • /home/keck/sch/runend.sch
  • weather: clear
  • windows and baffles: windows clear, little snow in and on the baffle

• 160723 05:47:49

  • cycleFridge enable=2, on all rx
  • checked connectors, PS, all looks good

• 160721 11:16:22 - 160723 05:47:49

  • /home/keck/sch/mce_autotune.sch
  • /data/keckdaq/autoqueue/pending/9_kCMB_50_064.sch
  • 160722 19:54:54 abort_schedule
  • powercycled syncbox, 5V, MCEs
  • 160722 20:01:41 Starting schedule: /home/keck/sch/mce_autotune.sch
  • 160722 20:06:33 Starting schedule: /data/keckdaq/autoqueue/pending/9_kCMB_50_064.sch
  • /home/keck/sch/runend.sch
  • weather: clear
  • windows and baffles: clear

• 160721 05:51:22

  • cycleFridge enable=2, on all rx
  • spudws froze
  • restarted gcp, masd

• 160719 11:25:16 - 160721 05:51:22

  • /home/keck/sch/mce_autotune.sch
  • /data/keckdaq/autoqueue/pending/9_kCMB_50_063.sch
  • 160719 13:09:36 abort_schedule
  • noise from mce0 still exists, power cycled MCEs, 12V, syncbox and MCE computers
  • restarted gcp, masd
  • 160719 13:22:08 Starting schedule: /home/keck/sch/mce_autotune.sch
  • 160719 13:25:26 Starting schedule: /data/keckdaq/autoqueue/pending/9_kCMB_50_063.sch
  • mediator crashed
  • 160719 16:21:41 abort_schedule
  • 160719 16:23:26 Starting schedule: /home/keck/sch/mce_autotune.sch
  • 160719 16:27:02 Starting schedule: /data/keckdaq/autoqueue/pending/9_kCMB_50_063.sch
  • masd4 stopped, had to restart keck37 and masd4
  • 160721 04:14:04 suspend_schedule
  • pmac tripped, fatal fol error on az, obvious reason
  • 160721 04:14:58 resume_schedule
  • /home/keck/sch/runend.sch
  • weather: clear
  • windows and baffles: clear

• 160719 05:59:39

  • cycleFridge enable=2, on all rx

• 160717 11:32:42 - 160719 05:59:39

  • /home/keck/sch/mce_autotune.sch
  • /data/keckdaq/autoqueue/pending/9_kCMB_50_062.sch
  • glycol pump switch without problems
  • /home/keck/sch/runend.sch
  • weather: hazy, blowing snow, wind dying, clearing
  • windows and baffles: windows clear, some snow in the baffles

• 160717 06:07:32

  • cycleFridge enable=2, on all rx
  • recharged comp 5
  • power cycled mce0
  • restarted masd

• 160715 11:48:39 - 160717 06:07:32

  • /home/keck/sch/mce_autotune.sch
  • /data/keckdaq/autoqueue/pending/9_kCMB_50_061.sch
  • /home/keck/sch/runend.sch
  • weather: haze, blowing snow, hazy
  • windows and baffles: windows clear, quite some snow on and in baffles

• 160715 06:29:23

  • cycleFridge enable=2, on all rx
  • el motor current driffting apart

• 160713 11:56:01 - 160715 06:15:23

  • /home/keck/sch/mce_autotune.sch
  • /data/keckdaq/autoqueue/pending/9_kCMB_50_060.sch
  • /home/keck/sch/runend.sch
  • weather: clear, haze, very hazy
  • windows and baffles: windows clear, some snow in the baffles

• 160713 06:36:06

  • cycleFridge enable=2, on all rx
  • recharged comp 1
  • el motor current driffting apart

• 160711 12:17:04 - 160713 06:23:16

  • /home/keck/sch/mce_autotune.sch
  • /data/keckdaq/autoqueue/pending/9_kCMB_50_059.sch
  • /home/keck/sch/runend.sch
  • weather: clear
  • windows and baffles: clear

• 160711 06:45:07

  • cycleFridge enable=2, on all rx

• 160709 11:57:16 - 160711 06:31:07

  • /home/keck/sch/mce_autotune.sch
  • /data/keckdaq/autoqueue/pending/9_kCMB_50_058.sch
  • no problems during pump switch
  • /home/keck/sch/runend.sch
  • weather: clear, very hazy, hazy, clearing
  • windows and baffles: clear

• 160709 06:38:57

  • cycleFridge enable=2, on all rx
  • 160709 08:17:49 Starting schedule: /home/keck/sch/starpoint_pole_keck_winter.sch
  • 160709 11:33:46 Exiting schedule: /home/keck/sch/starpoint_pole_keck_winter.sch

• 160707 12:08:54 - 160709 06:38:57

  • /home/keck/sch/mce_autotune.sch
  • /data/keckdaq/autoqueue/pending/9_kCMB_50_057.sch
  • keck34 unresponsive, restarted keck34 and masd1
  • 160707 12:19:05 Starting schedule: /home/keck/sch/mce_autotune.sch
  • /home/keck/sch/runend.sch
  • weather: hazy, clear
  • windows and baffles: clear

• 160707 06:48:02

  • cycleFridge enable=2, on all rx

• 160705 12:28:36 - 160707 06:48:02

  • /home/keck/sch/mce_autotune.sch
  • /data/keckdaq/autoqueue/pending/9_kCMB_50_056.sch
  • reset rx4 HK connectors
  • 160705 12:36:57 Starting schedule: /home/keck/sch/mce_autotune.sch
  • 160706 03:58:30 suspend_schedule
  • az diff jump, bad bearing in readhead roller 2B (which was replaced 2 weeks ago with a new one)
  • 160706 05:04:35 resume_schedule, data are good after resume schedule
  • /home/keck/sch/runend.sch
  • weather: clear, very hazy
  • windows and baffles: windows clear, some snow in the baffles, quite some snow on the baffles

• 160705 08:00

  • cycleFridge enable=2, on rx1,2,3,4
  • rx0 still cooling down slowly
  • recharged comp 2

• 160705 02:37:15

  • cycleFridge enable=2, on rx1,2,3,4
  • rx0 still cooling down slowly

• 160703 12:55:40 - 160703 22:07:15

  • /home/keck/sch/mce_autotune.sch
  • /data/keckdaq/autoqueue/pending/9_kCMB_50_055.sch
  • 160703 around 20:00 glycol pump failure, warm up of all rx
  • aborted run
  • rx0 and rx1, warmed up to about 40K, cooldown is very slow, pumped on rx0,rx1 and rx4
  • /home/keck/sch/runend.sch
  • weather: clear, cold :)
  • windows and baffles: clear

• 160703 07:02:34

  • cycleFridge enable=2, on all rx
  • recharged comp 4

• 160701 12:54:39 - 160703 07:02:34

  • /home/keck/sch/mce_autotune.sch
  • /data/keckdaq/autoqueue/pending/9_kCMB_50_054.sch
  • /home/keck/sch/runend.sch
  • weather: hazy, blowing snow, clearing, clear
  • windows and baffles: clear

• 160701 07:24:25

  • cycleFridge enable=2, on all rx

• 160629 12:43:21 - 160701 07:10:24

  • /home/keck/sch/mce_autotune.sch
  • /data/keckdaq/autoqueue/pending/9_kCMB_50_053.sch
  • /home/keck/sch/runend.sch
  • weather: blowing snow, hazy
  • windows and baffles: windows clear, quite some snow in the baffles

• 160629 07:18:16

  • cycleFridge enable=2, on all rx

• 160627 12:47:20 - 160629 07:18:16

  • /home/keck/sch/mce_autotune.sch
  • /data/keckdaq/autoqueue/pending/9_kCMB_50_052.sch
  • 160629 00:27:49 brown out
  • and another brown out with a 1 sec power outage, no scansets were missed
  • el motor currents drifting apart
  • 160629 01:47:31 imminent power outage, stopped telescope
  • 160629 01:50:18 stopped telescope. el motor current driffting
  • power outage did not happen
  • 160629 02:23:14 resume_schedule
  • missed one scanset
  • /home/keck/sch/runend.sch
  • weather: blowing snow, clear overhead, hazy
  • windows and baffles: windows clear, some snow in the baffles

• 160627 07:25:44

  • cycleFridge enable=2, on all rx

• 160625 12:58:02 - 160627 07:25:44

  • /home/keck/sch/mce_autotune.sch
  • /data/keckdaq/autoqueue/pending/9_kCMB_50_051.sch
  • 160625 16:40 keck36 was unresponsive, restarted keck36 and masd3
  • 160625 23:58:42 masd1 crashed
  • 160627 01:31:01 suspend_schedule, big az-diff jump, rehomed
  • 160627 01:37:47 resume_schedule
  • /home/keck/sch/runend.sch
  • weather: mostly clear, blowing snow
  • windows and baffles: windows clear, some snow in and on the baffles

• 160625 07:34:00

  • cycleFridge enable=2, on all rx
  • replaced read head roller 2B

• 160623 13:11:14 - 160625 07:34:00

  • /home/keck/sch/mce_autotune.sch
  • /data/keckdaq/autoqueue/pending/9_kCMB_50_050.sch
  • /home/keck/sch/runend.sch
  • weather: clear, mostly clear, blowing snow
  • windows and baffles: windows clear, some snow in the baffles

• 160623 07:41:51

  • cycleFridge enable=2, on all rx
  • spud ws1 froze
  • restarted gcp, masd

• 160621 13:12:55 - 160623 07:41:51

  • /home/keck/sch/mce_autotune.sch
  • /data/keckdaq/autoqueue/pending/9_kCMB_50_049.sch
  • /home/keck/sch/runend.sch
  • weather: clear
  • windows and baffles: windows clear, very little snow in the baffles

• 160621 07:49:43

  • cycleFridge enable=2, on all rx

• 160619 13:50:43 - 160621 07:49:43

  • /home/keck/sch/mce_autotune.sch
  • /data/keckdaq/autoqueue/pending/9_kCMB_50_048.sch
  • /home/keck/sch/runend.sch
  • weather: overcast, hazy, clearing, clear
  • windows and baffles: windows clear, some snow in the baffles

• 160619 07:57:37

  • cycleFridge enable=2, on all rx

• 160617 13:26:37 - 160619 07:57:37

  • /home/keck/sch/mce_autotune.sch
  • /data/keckdaq/autoqueue/pending/9_kCMB_50_047.sch
  • /home/keck/sch/runend.sch
  • weather: hazy, overcast, blowing snow, warm
  • windows and baffles: windows clear, lots of snow in the baffles

• 160617 08:05:28

  • cycleFridge enable=2, on all rx

• 160615 13:29:36 - 160617 08:05:28

  • /home/keck/sch/mce_autotune.sch
  • /data/keckdaq/autoqueue/pending/9_kCMB_50_046.sch
  • power outage 160616 about 10:00
  • power outage 160616 about 11:38
  • besides the missed scan sets no further problems due to the outagaes
  • /home/keck/sch/runend.sch
  • weather: hazy, blowing snow
  • windows and baffles: windows clear, very little snow on the baffles

• 160615 08:13:19

  • cycleFridge enable=2, on all rx
  • recharged comp 3

• 160613 13:38:38 - 160615 08:13:19

  • /home/keck/sch/mce_autotune.sch
  • /data/keckdaq/autoqueue/pending/9_kCMB_50_045.sch
  • /home/keck/sch/runend.sch
  • weather: clearing, clear, overcast, blowing snow, hazy
  • windows and baffles: windows clear, lots of snow in and on baffle

• 160613 08:21:11

  • cycleFridge enable=2, on all rx

• 160611 14:13:51 - 160613 08:21:11

  • /home/keck/sch/mce_autotune.sch
  • /data/keckdaq/autoqueue/pending/9_kCMB_50_044.sch
  • /home/keck/sch/runend.sch
  • weather: hazy, overcast, very warm, clearing
  • windows and baffles: windows clear, lots of snow in and on baffles

• 160611 08:29:03

  • cycleFridge enable=2, on all rx
  • recharged comp 1,2

• 160609 14:15:36 - 160611 08:29:03

  • /home/keck/sch/mce_autotune.sch
  • /data/keckdaq/autoqueue/pending/9_kCMB_50_043.sch
  • /home/keck/sch/runend.sch
  • weather: getting hazy, overcast, blowing snow, hazy
  • windows and baffles: windows clear, lots of snow in and on baffles

• 160609 08:50:56

  • cycleFridge enable=2, on all rx
  • el motor currents drifting apart

• 160607 14:13:30 - 160609 08:36:56

  • /home/keck/sch/mce_autotune.sch
  • /data/keckdaq/autoqueue/pending/9_kCMB_50_042.sch
  • /home/keck/sch/runend.sch
  • weather: clear, hazy, clear
  • windows and baffles: windows clear, very little snow in and on the baffles

• 160607 08:44:47

  • cycleFridge enable=2, on all rx
  • recharged comp 4
  • 160607 10:57:50 Starting schedule: /home/keck/sch/starpoint_pole_keck_winter.sch

• 160605 14:19:35 - 160607 08:44:47

  • /home/keck/sch/mce_autotune.sch
  • /data/keckdaq/autoqueue/pending/9_kCMB_50_041.sch
  • brief power outage 160605 22:00
  • el motor currents, driffting apart
  • switched signalhound power supply
  • /home/keck/sch/runend.sch
  • weather: clear
  • windows and baffles: clear

• 160605 08:52:39

  • cycleFridge enable=2, on all rx

• 160603 14:37:00 - 160605 08:52:39

  • /home/keck/sch/mce_autotune.sch
  • 160603 14:39:49 Starting schedule: /data/keckdaq/autoqueue/pending/1_keck_tcm_pidslow_test_001.sch
  • 160603 15:44:59 Starting schedule: /data/keckdaq/autoqueue/pending/9_kCMB_50_040.sch
  • /home/keck/sch/runend.sch
  • weather: overcast, blowing snow, clearing
  • windows and baffles: windows clear, lots of snow in the baffles

• 160603 09:14:32

  • cycleFridge enable=2, on all rx

• 160601 14:28:41 - 160603 09:00:31

  • /home/keck/sch/mce_autotune.sch
  • /data/keckdaq/autoqueue/pending/9_kCMB_50_039.sch
  • pager problems
  • aborted run
  • 160601 14:53:58 Starting schedule: /data/keckdaq/autoqueue/pending/9_kCMB_50_039.sch
  • /home/keck/sch/runend.sch
  • weather: hazy, overcast, blowing snow
  • windows and baffles: windows clear, lots of snow in and on the baffles

• 160601 09:08:22

  • cycleFridge enable=2, on all rx

• 160530 14:38:26 - 160601 09:08:22

  • /home/keck/sch/mce_autotune.sch
  • check_rms gave 2.5 for rx4, reseated the connectors
  • 160530 14:53:35 Starting schedule: /home/keck/sch/mce_autotune.sch
  • /data/keckdaq/autoqueue/pending/9_kCMB_50_038.sch
  • pmac tripped, problem with one of the back rollers, some piece of track got caught
  • after removal, it works fine again, roller rotates freely
  • el motor currents driffted apart
  • /home/keck/sch/runend.sch
  • weather: mostly clear, blowing snow, overcast
  • windows and baffles: windows clear, quite some snow in the baffles

• 160530 09:16:15

  • cycleFridge enable=2, on all rx
  • replaced bad He line with spare

• 160528 14:47:31 - 160530 09:16:15

  • /home/keck/sch/mce_autotune.sch
  • /data/keckdaq/autoqueue/pending/9_kCMB_50_037.sch
  • /home/keck/sch/runend.sch
  • weather: overcast, blowing snow, clearing
  • windows and baffles: windows clear, quite some snow in the baffles

• 160528 09:24:06

  • cycleFridge enable=2, on all rx
  • el motor current started to drift apart

• 160526 15:10:59 - 160528 09:24:06

  • /home/keck/sch/mce_autotune.sch
  • 160526 15:13:58 Starting schedule: /data/keckdaq/autoqueue/pending/1_keck_tcm_pid_test_002.sch
  • 160526 16:09:07 Starting schedule: /data/keckdaq/autoqueue/pending/9_kCMB_50_036.sch
  • /home/keck/sch/runend.sch
  • weather: mostly clear
  • windows and baffles: clear

• 160526 09:44:25

  • cycleFridge enable=2, on all rx
  • restarted spudws1, restarted gcp, masd

• 160524 15:13:25 - 160526 09:32:10

  • /home/keck/sch/mce_autotune.sch
  • /data/keckdaq/autoqueue/pending/9_kCMB_50_035.sch
  • He leak in A-Lo, lost 100psi over course of a few hours
  • rx0 was not working for reminder of run since the warm up to 350mk on the FPU
  • swapped A-Lo for spare
  • /home/keck/sch/runend.sch
  • weather: some clouds, mostly clear
  • windows and baffles: clear

• 160524 09:54:02

  • cycleFridge enable=2, on all rx
  • swapped bad He line for new spare

• 160522 15:26:37 - 160524 09:40:02

  • /home/keck/sch/mce_autotune.sch
  • /data/keckdaq/autoqueue/pending/9_kCMB_50_034.sch
  • /home/keck/sch/runend.sch
  • weather: haze, clearing, clouds, haze
  • windows and baffles: windows clear, some snow in the baffles

• 160522 10:01:28

  • cycleFridge enable=2, on all rx
  • lost 50 psi on comp 3, A Hi line leaked, was replaced with spare

• 160520 18:21:06 - 160522 09:47:53

  • /home/keck/sch/mce_autotune.sch
  • /data/keckdaq/autoqueue/pending/9_kCMB_50_033.sch
  • /home/keck/sch/runend.sch
  • weather: overcast, haze
  • windows and baffles: windows clear, some snow in the baffles

• 160520 13:04:23

  • spudws1 froze, didn't start fridgecycle, only realized it after the cleaing
  • restarted gcp, masd
  • cycleFridge enable=2, on all rx

• 160518 15:17:21 - 160520 09:55:20

  • /home/keck/sch/mce_autotune.sch
  • /data/keckdaq/autoqueue/pending/9_kCMB_50_032.sch
  • 160519 around 17:50 keck34 stopped recording, masd1 was still running
  • accidentally stopped masd2 as well for half a scanset
  • keck34 came back after reboot and restart of masd1
  • /home/keck/sch/runend.sch
  • weather: very hazy, overcast
  • windows and baffles: about a 3 to 4inch circle of snow on the center of windows, lots of snow in and on baffles

• 160518 10:03:38

  • cycleFridge enable=2, on all rx

• 160516 15:39:14 - 160518 10:03:38

  • /home/keck/sch/mce_autotune.sch
  • /data/keckdaq/autoqueue/pending/9_kCMB_50_031.sch
  • /home/keck/sch/runend.sch
  • weather: clear, cold
  • windows and baffles: clear

• 160516 10:16:41

  • cycleFridge enable=2, on all rx
  • el motor currents diverged
  • recharged comp 2,4

• 160514 15:54:39 - 160516 10:11:28

  • /home/keck/sch/mce_autotune.sch
  • /data/keckdaq/autoqueue/pending/9_kCMB_50_030.sch
  • /home/keck/sch/runend.sch
  • weather: clear, cold
  • windows and baffles: clear

• 160514 10:30:16

  • cycleFridge enable=2, on all rx

• 160512 15:46:00 - 160514 10:19:21

  • /home/keck/sch/mce_autotune.sch
  • 160512 15:48:54 Starting schedule: /data/keckdaq/autoqueue/pending/1_keck_tcm_pid_test_002.sch
  • 160512 16:54:03 Starting schedule: /data/keckdaq/autoqueue/pending/9_kCMB_50_029.sch
  • /home/keck/sch/runend.sch
  • weather: clear, cold
  • windows and baffles: clear

• 160512 10:27:12

  • cycleFridge enable=2, on all rx

• 160510 15:57:17 - 160512 10:27:12

  • /home/keck/sch/mce_autotune.sch
  • /data/keckdaq/autoqueue/pending/9_kCMB_50_028.sch
  • /home/keck/sch/runend.sch
  • weather: clear
  • windows and baffles: clear

• 160510 10:35:05

  • cycleFridge enable=2, on all rx

• 160508 16:14:14 - 160510 10:35:05

  • /home/keck/sch/mce_autotune.sch
  • /data/keckdaq/autoqueue/pending/9_kCMB_50_027.sch
  • /home/keck/sch/runend.sch
  • weather: clear, mostly clear
  • windows and baffles: windows clear, very little snow in the baffles

• 160508 10:54:55

  • cycleFridge enable=2, on all rx

• 160506 16:12:46 - 160508 10:42:58

  • /home/keck/sch/mce_autotune.sch
  • /data/keckdaq/autoqueue/pending/9_kCMB_50_026.sch
  • /home/keck/sch/runend.sch
  • weather: very hazy, blowing snow, overcast, warm, haze
  • windows and baffles: windows clear, some snow in the baffles

• 160506 10:51:16

  • cycleFridge enable=2, on all rx

• 160504 16:15:53 - 160506 10:50:46

  • /home/keck/sch/mce_autotune.sch
  • /data/keckdaq/autoqueue/pending/9_kCMB_50_025.sch
  • masd4 didn't record restarted masd4
  • 160504 16:22:12 schedule /home/keck/sch/mce_autotune.sch
  • 160504 16:23:26 Starting schedule: /data/keckdaq/autoqueue/pending/9_kCMB_50_025.sch
  • /home/keck/sch/runend.sch
  • weather: clear, bowing snow, very hazy
  • windows and baffles: windows clear, some snow in and on baffles

• 160504 10:58:39

  • cycleFridge enable=2, on all rx

• 160502 16:21:08 - 160504 10:58:39

  • /home/keck/sch/mce_autotune.sch
  • /data/keckdaq/autoqueue/pending/9_kCMB_50_024.sch
  • /home/keck/sch/runend.sch
  • weather: mostly clear, some blowing snow
  • windows and baffles: windows clear, very little snow in the baffles

• • cycleFridge enable=2, on all rx

• 160430 16:48:45 -

  • /home/keck/sch/mce_autotune.sch
  • 160430 16:52:57 Starting schedule: /data/keckdaq/autoqueue/pending/1_keck_tcm_pid_test.sch
  • 160430 17:23:06 Starting schedule: /data/keckdaq/autoqueue/pending/9_kCMB_50_023.sch
  • /home/keck/sch/runend.sch
  • weather: clear, some haze, clearing
  • windows and baffles: windows clear, very little snow in the baffles

• 160430 11:28:23

  • cycleFridge enable=2, on all rx

• 160428 17:22:28 - 160430 11:14:23

  • /home/keck/sch/mce_autotune.sch
  • /data/keckdaq/autoqueue/pending/9_kCMB_50_022.sch
  • /home/keck/sch/runend.sch
  • weather: clear, haze, clear
  • windows and baffles: windows clear, very little snow in the baffles

• 160428 11:36:16

  • cycleFridge enable=2, on all rx
  • recharged compressor 1,4
  • spudws froze, restarted gcp, masd
  • problems with masd4, restarted keck37 and masd4

• 160426 17:04:27 - 160428 11:22:16

  • /home/keck/sch/mce_autotune.sch
  • /data/keckdaq/autoqueue/pending/9_kCMB_50_021.sch
  • /home/keck/sch/runend.sch
  • weather: clear
  • windows and baffles: clear

• 160426 11:44:09

  • cycleFridge enable=2, on all rx
  • recharged compressor 2

• 160424 16:59:07 - 160426 11:30:09

  • /home/keck/sch/mce_autotune.sch
  • /data/keckdaq/autoqueue/pending/9_kCMB_50_020.sch
  • /home/keck/sch/runend.sch
  • weather: clear
  • windows and baffles: clear

• 160424 11:37:59

  • cycleFridge enable=2, on all rx

• 160422 17:06:11 - 160424 11:37:59

  • /home/keck/sch/mce_autotune.sch
  • /data/keckdaq/autoqueue/pending/9_kCMB_50_019.sch
  • /home/keck/sch/runend.sch
  • weather: hazy and clouds, clear, hazy
  • windows and baffles: windows clear, a little snow in the baffles

• 160422 11:45:52

  • cycleFridge enable=2, on all rx

• 160420 18:21:50 - 160422 11:45:52

  • /home/keck/sch/mce_autotune.sch
  • /data/keckdaq/autoqueue/pending/9_kCMB_50_018.sch
  • /home/keck/sch/runend.sch
  • weather: clear, haze, clouds, blowing snow
  • windows and baffles: windows clear, lots of snow in the baffles

• 160420 11:53:43

  • cycleFridge enable=2, on all rx
  • recharged compressor 3
  • 160420 14:45:34 Starting schedule: /home/keck/sch/starpoint_pole_keck_winter.sch
  • 160420 18:21:50 Exiting schedule: /home/keck/sch/starpoint_pole_keck_winter.sch

• 160418 17:39:27 - 160420 11:53:43

  • /home/keck/sch/mce_autotune.sch
  • /data/keckdaq/autoqueue/pending/9_kCMB_50_017.sch
  • /home/keck/sch/runend.sch
  • weather: clear
  • windows and baffles: clear

• 160418 12:15:35

  • cycleFridge enable=2, on all rx

• 160416 17:30:43 - 160418 12:01:35

  • /home/keck/sch/mce_autotune.sch
  • data/keckdaq/autoqueue/pending/1_knoise_responsivity_9_013.sch‎
  • 160416 18:12:21 abort_schedule
  • clouds were moving in, was perfect until then
  • 160416 18:12:54 Starting schedule: /data/keckdaq/autoqueue/pending/9_kCMB_50_016.sch
  • /home/keck/sch/runend.sch
  • weather:clear, mostly clear, clear
  • windows and baffles: clear

• 160416 12:09:00

  • cycleFridge enable=2, on all rx

• 160414 17:36:13 - 160416 12:09:00

  • /home/keck/sch/mce_autotune.sch
  • /data/keckdaq/autoqueue/pending/9_kCMB_50_015.sch
  • masd4 stopped for a few scansets in the last set of 10
  • restarted keck37 and masd4
  • /home/keck/sch/runend.sch
  • weather: totally overcast, clearing clear
  • windows and baffles: windows clear, lots of snow in and on baffles

• 160414 12:17:17

  • cycleFridge enable=2, on all rx

• 160412 17:49:43 - 160414 12:17:17

  • /home/keck/sch/mce_autotune.sch
  • /data/keckdaq/autoqueue/pending/1_knoise_responsivity_9_012.sch‎
  • 160412 23:58:45 Starting schedule: /data/keckdaq/autoqueue/pending/9_kCMB_50_014.sch
  • /home/keck/sch/runend.sch
  • weather: clear, clouds, blowing snow
  • windows and baffles: windows clear, little snow on the baffles

• 160412 12:25:09

  • cycleFridge enable=2, on all rx

• 160410 17:51:41 - 160412 12:25:09

  • /home/keck/sch/mce_autotune.sch
  • did some pager testing
  • 160410 18:00:09 Starting schedule: /data/keckdaq/autoqueue/pending/9_kCMB_50_013.sch
  • /home/keck/sch/runend.sch
  • weather: thin clouds, blowing snow, clearing
  • windows and baffles: windows clear, quite some snow in the baffles

• 160410 12:33:02

  • cycleFridge enable=2, on all rx

• 160408 17:53:35 - 160410 12:33:02

  • /home/keck/sch/mce_autotune.sch
  • /data/keckdaq/autoqueue/pending/9_kCMB_50_012.sch
  • /home/keck/sch/runend.sch
  • weather: overcast, haze
  • windows and baffles: windows clear, quite some snow in and on baffle

• 160408 12:41:24

  • cycleFridge enable=2, on all rx
  • after homing el motor current started to drift apart, opened loops and closed them again

• 160406 18:04:06 - 160408 12:41:24

  • /home/keck/sch/mce_autotune.sch
  • /data/keckdaq/autoqueue/pending/9_kCMB_50_011.sch
  • 160407 23:30 radio usage in MAPO due to a flase fire alarm
  • 160408 04:11 started messing with gps card
  • 160408 06:12 gps update successful, gps reading correct time
  • run stopped with wrong time, but also did not continue after time was fixed
  • 160408 06:59:54 abort_schedule
  • had to restart gcp and masd
  • 160408 07:08:11 Starting schedule: /data/keckdaq/autoqueue/pending/9_kCMB_50_011.sch
  • 160408 12:38:46 abort_schedule
  • /home/keck/sch/runend.sch
  • weather: mostly clear, getting hazy, wind picking up again
  • windows and baffles: windows clear, quite some snow in and on the baffles

• 160406 12:48:19

  • cycleFridge enable=2, on all rx

• 160404 18:43:30 - 160406 12:48:19

  • /home/keck/sch/mce_autotune.sch
  • /data/keckdaq/autoqueue/pending/9_kCMB_50_010.sch
  • 160405 about 20:00 keck36 bolos stopped, masd3 was still writing frames
  • restarted keck36 and masd3, worked afterwards again
  • /home/keck/sch/runend.sch
  • weather: blowing snow, overcast, clearing, mostly clear
  • windows and baffles: windows clear, quite some snow in and on baffles

• 160404 12:56:36

  • cycleFridge enable=2, on all rx

• 160402 18:25:45 - 160404 12:56:36

  • /home/keck/sch/mce_autotune.sch
  • /data/keckdaq/autoqueue/pending/9_kCMB_50_009.sch
  • /home/keck/sch/runend.sch
  • weather: haze, blowing snow, overcast
  • windows and baffles:

• 160402 13:04:28

  • cycleFridge enable=2, on all rx
  • recharged comp 2,4
  • lowered the "wait all keck" time from 120s to 90s as Colin instructed

• 160331 19:02:53 - 160402 13:04:28

  • /home/keck/sch/mce_autotune.sch
  • /data/keckdaq/autoqueue/pending/9_kCMB_50_008.sch
  • 160331 19:14:10 abort_schedule
  • did some pager tests
  • 160331 19:14:56 Starting schedule: /data/keckdaq/autoqueue/pending/9_kCMB_50_008.sch
  • several missed scansets
  • /home/keck/sch/runend.sch
  • weather: clear overhead, some blowing snow, getting hazy, some perticipation
  • windows and baffles: windows clear, very little snow in the baffles

• 160331 13:45:31

  • cycleFridge enable=2, on all rx
  • el motor currents drifting apart
  • spudws1 died, replaced with spudws2

• 160329 18:43:28 - 160331 13:12:21

  • /home/keck/sch/mce_autotune.sch
  • /data/keckdaq/autoqueue/pending/9_kCMB_50_007.sch
  • about 160329 19:00 MAPO signalhound should work with proper power again
  • 160330 13:13:45 abort_schedule
  • updated TES biases for rx0, rx2
  • 160330 13:13:45 Starting schedule: /home/keck/gcp/control/sch/config_hw.sch
  • 160330 13:14:54 Starting schedule: /data/keckdaq/autoqueue/pending/9_kCMB_50_007.sch
  • missed scanset again LST 1900 on second half
  • spudws1 has some problems
  • /home/keck/sch/runend.sch
  • weather: clear, blowing snow, but clear overhead
  • windows and baffles: windows clear, some snow in the baffles

• 160329 13:25:38

  • cycleFridge enable=2, on all rx
  • restarted gcp, masd to have windows back on spudws1

• 160327 18:44:06 - 160329 13:20:11

  • /home/keck/sch/mce_autotune.sch
  • /data/keckdaq/autoqueue/pending/9_kCMB_50_006.sch
  • daq froze up
  • 160328 07:36:00 abort_schedule
  • restarted gcp, masd
  • 160328 07:38:24 Starting schedule: /home/keck/sch/mce_autotune.sch
  • 160328 07:39:39 Starting schedule: /data/keckdaq/autoqueue/pending/9_kCMB_50_006.sch
  • furnace in MAPO quit, building cooled substantially
  • /home/keck/sch/runend.sch
  • weather:little hazy, clear
  • windows and baffles: clear

• 160327 13:28:03

  • cycleFridge enable=2, on all rx
  • recharged comp 3

• 160325 19:07:30 - 160327 13:28:03

  • /home/keck/sch/mce_autotune.sch
  • /data/keckdaq/autoqueue/pending/1_knoise_responsivity_9_011.sch‎
  • 160325 19:11:11 abort_schedule
  • masd4 gives problems (every run except 004 !?)
  • restart mce power, gcp and masd
  • still problems
  • rebooted keck37
  • had to abort schedule several times
  • 160325 19:33:05 Starting schedule: /home/keck/sch/mce_autotune.sch
  • 160325 19:34:27 Starting schedule: /data/keckdaq/autoqueue/pending/1_knoise_responsivity_9_011.sch‎
  • one scanset from 160327 12:10 lst is missing ??? no obvious problem with telescope
  • /home/keck/sch/runend.sch
  • weather: thin clouds,clearing, clear
  • windows and baffles: clear

• 160325 13:48:49

  • cycleFridge enable=2, on all rx
  • recharged comp 1
  • 160325 16:34:15 Starting schedule: /home/keck/sch/starpoint_pole_keck_summer.sch

• 160323 19:07:01 - 160325 13:35:49

  • /home/keck/sch/mce_autotune.sch
  • /data/keckdaq/autoqueue/pending/9_kCMB_50_004.sch
  • /home/keck/sch/runend.sch
  • weather: clear, thin clouds, haze
  • windows and baffles: windows clear, very little snow on the baffles

• 160323 13:46:37

  • cycleFridge enable=2, on all rx

• 160321 19:37:10 - 160323 13:43:40

  • /home/keck/sch/mce_autotune.sch
  • /data/keckdaq/autoqueue/pending/9_kCMB_50_003.sch
  • no sample bolos on rx4, although masd4 is running
  • 160321 19:40:58 abort_schedule
  • restarted sync box, gcp. masd
  • 160321 19:50:16 Starting schedule: /home/keck/sch/mce_autotune.sch
  • 160321 19:53:33 Starting schedule: /data/keckdaq/autoqueue/pending/9_kCMB_50_003.sch
  • /home/keck/sch/runend.sch
  • tried run 004, but it exited as well after the pager commands
  • weather: blowing snow, overcast, clearing , clear
  • windows and baffles: windows clear, very little snow in the baffles

• 160321 14:21:34

  • cycleFridge enable=2, on all rx

• 160319 19:31:48 - 160321 13:51:39

  • /home/keck/sch/mce_autotune.sch
  • /data/keckdaq/autoqueue/pending/9_kCMB_50_002.sch
  • schedule did exit again
  • strange problems with masd4
  • between 160319 19:31:48 and 160319 20:31:48 muliple restarts with different schedules

  • 160319 19:31:48 Starting schedule: /home/keck/sch/mce_autotune.sch
    160319 19:33:02 Exiting schedule: /home/keck/sch/mce_autotune.sch
    160319 19:35:52 Starting schedule: /data/keckdaq/autoqueue/pending/9_kCMB_50_002.sch
    160319 19:37:07 Exiting schedule: /data/keckdaq/autoqueue/pending/9_kCMB_50_002.sch
    160319 19:37:07 Moving completed autoqueue schedule to done directory
    160319 19:37:07 moving /data/keckdaq/autoqueue/pending/9_kCMB_50_002.sch to /data/keckdaq/autoqueue/done/9_kCMB_50_002.sch
    160319 19:44:58 Starting schedule: /data/keckdaq/autoqueue/pending/9_kCMB_50_002.sch
    160319 19:46:12 Exiting schedule: /data/keckdaq/autoqueue/pending/9_kCMB_50_002.sch
    160319 19:46:12 Moving completed autoqueue schedule to done directory
    160319 19:46:12 moving /data/keckdaq/autoqueue/pending/9_kCMB_50_002.sch to /data/keckdaq/autoqueue/done/9_kCMB_50_002.sch
    160319 19:49:52 Starting schedule: /data/keckdaq/autoqueue/pending/9_kCMB_50_002.sch
    160319 19:51:09 Exiting schedule: /data/keckdaq/autoqueue/pending/9_kCMB_50_002.sch
    160319 19:51:09 Moving completed autoqueue schedule to done directory
    160319 19:51:09 moving /data/keckdaq/autoqueue/pending/9_kCMB_50_002.sch to /data/keckdaq/autoqueue/done/9_kCMB_50_002.sch
    160319 19:51:43 Starting schedule: /data/keckdaq/autoqueue/pending/9_kCMB_50_000.sch
    160319 19:53:38 abort_schedule
    160319 19:53:38 Exiting schedule: /data/keckdaq/autoqueue/pending/9_kCMB_50_000.sch
    160319 20:05:15 schedule /home/keck/sch/mce_autotune.sch
    160319 20:05:15 Starting schedule: /home/keck/sch/mce_autotune.sch
    160319 20:06:30 Exiting schedule: /home/keck/sch/mce_autotune.sch
    160319 20:07:26 Starting schedule: /data/keckdaq/autoqueue/pending/9_kCMB_50_002.sch
    160319 20:08:41 Exiting schedule: /data/keckdaq/autoqueue/pending/9_kCMB_50_002.sch
    160319 20:08:41 Moving completed autoqueue schedule to done directory
    160319 20:08:41 moving /data/keckdaq/autoqueue/pending/9_kCMB_50_002.sch to /data/keckdaq/autoqueue/done/9_kCMB_50_002.sch
    160319 20:09:56 Exiting schedule: /home/keck/gcp/control/conf/keck/control.init
    160319 20:09:59 Starting schedule: /home/keck/gcp/control/conf/keck/mediator.init
    160319 20:09:59 Exiting schedule: /home/keck/gcp/control/conf/keck/mediator.init
    160319 20:11:18 Starting schedule: /data/keckdaq/autoqueue/pending/9_kCMB_50_001.sch
    160319 20:13:08 abort_schedule
    160319 20:13:08 Exiting schedule: /data/keckdaq/autoqueue/pending/9_kCMB_50_001.sch
    160319 20:23:24 Exiting schedule: /home/keck/gcp/control/conf/keck/control.init
    160319 20:23:27 Starting schedule: /home/keck/gcp/control/conf/keck/mediator.init
    160319 20:23:27 Exiting schedule: /home/keck/gcp/control/conf/keck/mediator.init
    160319 20:29:20 schedule /home/keck/sch/mce_autotune.sch
    160319 20:29:20 Starting schedule: /home/keck/sch/mce_autotune.sch
    160319 20:30:38 Exiting schedule: /home/keck/sch/mce_autotune.sch
    

  • trying to figure out what is wrong with masd4 and why schedules crash
  • will miss the first scanset !!
  • 160319 20:31:48 Starting schedule: /home/keck/sch/mce_autotune.sch
  • 160319 20:39:30 Starting schedule: /data/keckdaq/autoqueue/pending/9_kCMB_50_002.sch
  • restarted gcp, masd & sync box, masd4 is back online
  • commented all the pager lines out of the schedule, this time it worked !??
  • 160320 05:11 pmac stopped
  • pager did not go off, did some test, but pager not working for this run
  • 160321 13:52:45 to 160321 14:21:34
  • testing of cmb schedules
  • /home/keck/sch/runend.sch
  • weather: clear, wind picking up, blowing snow, overcast
  • windows and baffles:

• 160319 14:12:22

  • cycleFridge enable=2, on all rx

• 160317 19:25:24 - 160319 13:59:22

  • /home/keck/sch/mce_autotune.sch
  • /data/keckdaq/autoqueue/pending/9_kCMB_50_001.sch
  • /home/keck/sch/runend.sch
  • weather: clear
  • windows and baffles: clear

• 160317 14:14:04

  • cycleFridge enable=2, on all rx

• 160315 18:22:13 - 160317 14:07:16

  • /home/keck/sch/mce_autotune.sch
  • /data/keckdaq/autoqueue/pending/9_kCMB_50_000.sch
  • schedule stopped twice due to pager issues, commented out pager lines
  • 160315 18:38:52 Starting schedule: /data/keckdaq/autoqueue/pending/9_kCMB_50_000.sch
  • 160315 18:56:44 abort_schedule
  • was still going for the last gallactic plane scan
  • 160315 20:00:50 Starting schedule: /home/keck/sch/mce_autotune.sch
  • 160315 20:02:13 Starting schedule: /data/keckdaq/autoqueue/pending/9_kCMB_50_000.sch
  • rx0 baffle temp sensor reads high, it is about 30K over actual temperature
  • /home/keck/sch/runend.sch
  • weather: clear
  • windows and baffles: clear

---

CMB 2016

---

• 160315 12:37:29

  • cycleFridge enable=2, on all rx

• 160315 05:02:08

  • cycleFridge enable=2, on all rx
  • 160315 08:38:22 Starting schedule: /home/keck/sch/starpoint_pole_keck_summer.sch

• 160314 04:35:53

  • removed FFF
  • compressors were off for less than 15min, rx0 warmed up most, to 25K on the FPU, was the only
  • rx I had to pump on for 10 min, all cooling now nicely
  • cycleFridge enable=4, on all rx
  • 160314 19:34:47 restarted masd and gcp

• 160313 08:29:40

  • cycleFridge enable=2, on all rx

• 160313 00:47:30 - 160313 08:29:40

  • /home/keck/sch/mce_autotune.sch
  • /data/keckdaq/autoqueue/pending/7_ffflat_dslmast_31c_016.sch
  • weather: mostly clear, haze on horizon

• 160312 19:45:20

  • cycleFridge enable=2, on all rx

• 160312 12:02:40 - 160312 19:45:19

  • /home/keck/sch/mce_autotune.sch
  • /data/keckdaq/autoqueue/pending/7_ffflat_dslmast_32d_016.sch
  • weather: mostly clear, haze on horizon

• 160312 07:11:20

  • cycleFridge enable=2, on all rx

• 160311 23:28:58 - 160312 07:11:19

  • /home/keck/sch/mce_autotune.sch
  • /data/keckdaq/autoqueue/pending/7_ffflat_dslmast_32b_016.sch
  • weather: partly overcast, haze

• 160311 18:34:01

  • cycleFridge enable=2, on all rx

• 160311 10:51:44 - 160311 18:34:00

  • /home/keck/sch/mce_autotune.sch
  • /data/keckdaq/autoqueue/pending/7_ffflat_dslmast_31a_016.sch
  • weather: haze

• 160311 05:37:01

  • cycleFridge enable=2, on all rx

• 160310 21:54:32 - 160311 05:37:01

  • /home/keck/sch/mce_autotune.sch
  • /data/keckdaq/autoqueue/pending/7_ffflat_dslmast_32i_015.sch
  • weather: haze

• 160310 15:26:36

  • cycleFridge enable=2, on all rx

• 160310 07:44:30 - 160310 15:26:36

  • /home/keck/sch/mce_autotune.sch
  • /data/keckdaq/autoqueue/pending/7_ffflat_dslmast_31j_015.sch
  • weather: clear

• 160310 02:42:20

  • cycleFridge enable=2, on all rx

• 160309 18:59:07 - 160310 02:42:20

  • /home/keck/sch/mce_autotune.sch
  • /data/keckdaq/autoqueue/pending/7_ffflat_dslmast_31h_015.sch
  • weather: clear

• 160309 13:45:56

  • cycleFridge enable=2, on all rx

• 160309 06:03:33 - 160309 13:45:56

  • /home/keck/sch/mce_autotune.sch
  • /data/keckdaq/autoqueue/pending/7_ffflat_dslmast_31g_015.sch
  • weather: clear

• 160309 00:29:31

  • spudws 1 froze, restarted gcp, masd
  • cycleFridge enable=2, on all rx

• 160308 16:30:00 - 160309 00:11:06

  • /home/keck/sch/mce_autotune.sch
  • /data/keckdaq/autoqueue/pending/7_ffflat_dslmast_32f_016.sch
  • weather: clear

• 160308 10:04:07

  • cycleFridge enable=2, on all rx

• 160308 04:10:32 - 160308 10:03:12

  • /home/keck/sch/mce_autotune.sch
  • /data/keckdaq/autoqueue/pending/7_ffflat_dslmast_32f_015.sch
  • 160308 10:03:12 abort_schedule
  • masd 3 stopped, would have probably been ok for this run due to the rx that are under the mirror
  • weather: mostly clear

• 160307 22:50:27

  • cycleFridge enable=2, on all rx

• 160307 15:08:42 - 160307 22:50:26

  • /home/keck/sch/mce_autotune.sch
  • /data/keckdaq/autoqueue/pending/7_ffflat_dslmast_31e_015.sch
  • weather: mostly clear, getting colder

• • cycleFridge enable=2, on all rx

• 160307 02:20:54 - 160307 10:04:06

  • /home/keck/sch/mce_autotune.sch
  • /data/keckdaq/autoqueue/pending/7_ffflat_dslmast_32d_015.sch
  • weather: mostly clear

• 160306 20:23:49

  • cycleFridge enable=2, on all rx

• 160306 12:41:38 - 160306 20:23:48

  • /home/keck/sch/mce_autotune.sch
  • /data/keckdaq/autoqueue/pending/7_ffflat_dslmast_31c_015.sch
  • weather: mostly clear

• 160306 07:42:57

  • cycleFridge enable=2, on all rx

• 160306 00:01:19 - 160306 07:42:57

  • /home/keck/sch/mce_autotune.sch
  • /data/keckdaq/autoqueue/pending/7_ffflat_dslmast_32b_015.sch
  • weather: mostly clear

• 160305 19:01:17

  • cycleFridge enable=2, on all rx

• 160305 11:18:54 - 160305 19:01:16

  • /home/keck/sch/mce_autotune.sch
  • /data/keckdaq/autoqueue/pending/7_ffflat_dslmast_31a_015.sch
  • weather: mostly clear

• 160305 06:13:26

  • cycleFridge enable=2, on all rx

• 160304 22:31:33 - 160305 06:13:26

  • /home/keck/sch/mce_autotune.sch
  • /data/keckdaq/autoqueue/pending/7_ffflat_dslmast_31j_014.sch
  • weather: clear, some haze on horizon

• 160304 14:41:19

  • cycleFridge enable=2, on all rx

• 160304 06:58:49 - 160304 14:41:18

  • /home/keck/sch/mce_autotune.sch
  • /data/keckdaq/autoqueue/pending/7_ffflat_dslmast_32i_014.sch
  • weather: clear, some clouds on horizon

• 160304 01:43:13

  • cycleFridge enable=2, on all rx
  • recharged compressor 2 and 4

• 160303 18:00:00 - 160304 01:43:13

  • /home/keck/sch/mce_autotune.sch
  • /data/keckdaq/autoqueue/pending/7_ffflat_dslmast_31h_014.sch
  • weather: clear, some clouds on horizon

• 160303 11:58:41

  • cycleFridge enable=2, on all rx

• 160303 04:17:09 - 160303 11:58:41

  • /home/keck/sch/mce_autotune.sch
  • /data/keckdaq/autoqueue/pending/7_ffflat_dslmast_31g_014.sch
  • weather: haze on horizon, clear above

• 160302 22:41:49

  • cycleFridge enable=2, on all rx

• 160302 15:00:00 - 160302 22:41:49

  • /home/keck/sch/mce_autotune.sch
  • /data/keckdaq/autoqueue/pending/7_ffflat_dslmast_32f_014.sch
  • weather: haze on horizon, clear above

• 160302 08:50:35

  • cycleFridge enable=2, on all rx

• 160302 01:08:21 - 160302 08:50:35

  • /home/keck/sch/mce_autotune.sch
  • /data/keckdaq/autoqueue/pending/7_ffflat_dslmast_31e_014.sch
  • weather: some haze on the horizon

• 160301 19:17:53

  • cycleFridge enable=2, on all rx

• 160301 10:25:42 - 160301 18:10:01

  • /home/keck/sch/mce_autotune.sch
  • /data/keckdaq/autoqueue/pending/7_ffflat_dslmast_32d_014.sch
  • weather: overcast, hazy

• 160301 05:18:14

  • cycleFridge enable=2, on all rx

• 160229 21:36:28 - 160301 05:18:14

  • /home/keck/sch/mce_autotune.sch
  • /data/keckdaq/autoqueue/pending/7_ffflat_dslmast_31c_014.sch
  • weather: overcast, hazy, warm

• 160229 16:30:47

  • cycleFridge enable=2, on all rx

• 160229 08:49:14 - 160229 16:30:46

  • /home/keck/sch/mce_autotune.sch
  • /data/keckdaq/autoqueue/pending/7_ffflat_dslmast_32b_014.sch
  • weather: overcast, hazy, warm

• 160229 03:38:23

  • cycleFridge enable=2, on all rx

• 160228 19:56:32 - 160229 03:38:23

  • moved FFF to forward position
  • A,B @ 15 in
  • E,F @ 55.5 in
  • G,H @ 29 in
  • /home/keck/sch/mce_autotune.sch
  • /data/keckdaq/autoqueue/pending/7_ffflat_dslmast_31a_014.sch
  • weather: clear, some haze on the horizon

• 160228 11:46:43

  • cycleFridge enable=2, on all rx

• 160228 02:48:57

  • cycleFridge enable=2, on all rx

• 160227 23:05:17

  • cycleFridge enable=2, on all rx

• 160227 11:50:54

  • cycleFridge enable=2, on all rx

• 160227 04:04:07 - 160227 11:50:53

  • didn't do mce_autotune since rx4 only cooled down to 350mk
  • /data/keckdaq/autoqueue/pending/7_ffflat_dslmast_32f_013.sch
  • 160227 04:06:01 abort_schedule
  • keck37 crashed which was probably the reason for slightly elevated temp on rx4
  • 160227 04:10:55 Starting schedule: /data/keckdaq/autoqueue/pending/7_ffflat_dslmast_32f_013.sch
  • weather: blowing snow

• 160226 22:00:09

  • cycleFridge enable=2, on all rx

• 160226 09:03:45 - 160226 21:02:01

  • /home/keck/sch/mce_autotune.sch
  • /data/keckdaq/autoqueue/pending/7_ffflat_dslmast_32f_013.sch
  • Telescope wasn't moving, but no errors were displayed
  • 160226 12:58:04 abort_schedul
  • rehomed, no problems, restarted gcp and masd
  • 160226 13:08:47 Starting schedule: /data/keckdaq/autoqueue/pending/7_ffflat_dslmast_32f_013.sch
  • still would not track
  • 160226 13:14:13 abort_schedule
  • 160226 13:15:02 Starting schedule: /data/keckdaq/autoqueue/pending/7_ffflat_dslmast_31e_013.sch
  • 160226 13:18:39 abort_schedule
  • slewing would work by hand
  • 160226 13:21:41 Starting schedule: /data/keckdaq/autoqueue/pending/7_ffflat_dslmast_31e_013.sch
  • now it works again ???
  • weather: blowing snow

• 160226 03:52:19

  • cycleFridge enable=2, on all rx

• 160225 20:09:36 - 160226 03:52:19

  • /home/keck/sch/mce_autotune.sch
  • /data/keckdaq/autoqueue/pending/7_ffflat_dslmast_32d_013.sch
  • weather: blowing snow

• 160225 10:32:05

  • cycleFridge enable=2, on all rx

• 160225 02:49:53 - 160225 10:32:05

  • /home/keck/sch/mce_autotune.sch
  • /data/keckdaq/autoqueue/pending/7_ffflat_dslmast_32b_013.sch
  • weather: blowing snow

• 160224 21:42:25

  • cycleFridge enable=2, on all rx

• 160224 14:00:00 - 160224 21:42:25

  • /home/keck/sch/mce_autotune.sch
  • /data/keckdaq/autoqueue/pending/7_ffflat_dslmast_31j_013.sch
  • weather: blowing snow

• 160224 07:58:54

  • cycleFridge enable=2, on all rx

• 160224 00:16:48 - 160224 07:58:54

  • /home/keck/sch/mce_autotune.sch
  • /data/keckdaq/autoqueue/pending/7_ffflat_dslmast_31c_013.sch
  • weather: blowing snow

• 160223 18:19:07

  • cycleFridge enable=2, on all rx

• 160223 10:37:15 - 160223 18:19:07

  • /home/keck/sch/mce_autotune.sch
  • /data/keckdaq/autoqueue/pending/7_ffflat_dslmast_31a_013.sch
  • weather: blowing snow

• 160223 05:32:15

  • cycleFridge enable=2, on all rx

• 160222 21:50:42 - 160223 05:32:15

  • /home/keck/sch/mce_autotune.sch
  • /data/keckdaq/autoqueue/pending/7_ffflat_dslmast_32i_012.sch
  • weather: blowing snow

• 160222 13:55:35

  • cycleFridge enable=2, on all rx

• 160222 06:12:20 - 160222 13:55:35

  • /home/keck/sch/mce_autotune.sch
  • /data/keckdaq/autoqueue/pending/7_ffflat_dslmast_32h_012.sch
  • weather: clear, clouds on horizon

• 160222 00:36:30

  • cycleFridge enable=2, on all rx

• 160221 16:54:23 - 160222 00:36:29

  • /home/keck/sch/mce_autotune.sch
  • /data/keckdaq/autoqueue/pending/7_ffflat_dslmast_32g_012.sch
  • weather: clearing

• 160221 10:52:48

  • cycleFridge enable=2, on all rx

• 160221 03:09:57 - 160221 10:52:48

  • /home/keck/sch/mce_autotune.sch
  • /data/keckdaq/autoqueue/pending/7_ffflat_dslmast_32f_012.sch
  • weather: hazy, overcast

• 160220 22:13:58

  • cycleFridge enable=2, on all rx

• 160220 14:32:03 - 160220 22:13:58

  • /home/keck/sch/mce_autotune.sch
  • /data/keckdaq/autoqueue/pending/7_ffflat_dslmast_32d_012.sch
  • weather: hazy, overcast

• 160220 09:28:14

  • cycleFridge enable=2, on all rx
  • recharged compressor 1,3

• 160220 01:30:40 - 160220 09:28:14

  • /home/keck/sch/mce_autotune.sch
  • /data/keckdaq/autoqueue/pending/7_ffflat_dslmast_31j_012.sch
  • weather: clear, some clouds on the horizon

• 160219 19:51:48

  • cycleFridge enable=2, on all rx

• 160219 12:10:16 - 160219 19:51:47

  • /home/keck/sch/mce_autotune.sch
  • /data/keckdaq/autoqueue/pending/7_ffflat_dslmast_32b_012.sch
  • weather: clear, few clouds on horizon

• 160219 06:19:26

  • cycleFridge enable=2, on all rx

• 160218 22:37:17 - 160219 06:19:25

  • /home/keck/sch/mce_autotune.sch
  • /data/keckdaq/autoqueue/pending/7_ffflat_dslmast_31c_012.sch
  • weather: clear, few clouds on horizon

• 160218 16:58:30

  • cycleFridge enable=2, on all rx

• 160218 09:16:38 - 160218 16:58:29

  • /home/keck/sch/mce_autotune.sch
  • /data/keckdaq/autoqueue/pending/7_ffflat_dslmast_31a_012.sch
  • weather: clear, clouds on the horizon

• 160218 04:07:04

  • cycleFridge enable=2, on all rx

• 160217 20:24:24 - 160218 04:07:04

  • /home/keck/sch/mce_autotune.sch
  • /data/keckdaq/autoqueue/pending/7_ffflat_dslmast_32i_011.sch
  • weather: mostly clear

• 160217 14:15:14

  • cycleFridge enable=2, on all rx

• 160217 06:33:43 - 160217 14:15:14

  • /home/keck/sch/mce_autotune.sch
  • /data/keckdaq/autoqueue/pending/7_ffflat_dslmast_32g_011.sch
  • weather: overcast, clearing

• 160217 01:25:11

  • cycleFridge enable=2, on all rx

• 160216 17:43:57 - 160217 01:25:11

  • /data/keckdaq/autoqueue/pending/7_ffflat_dslmast_32f_011.sch
  • weather: partly cloudy

• 160216 12:00:28

  • cycleFridge enable=2, on all rx

• 160216 04:18:29 - 160216 12:00:28

  • /home/keck/sch/mce_autotune.sch
  • /data/keckdaq/autoqueue/pending/7_ffflat_dslmast_32d_011.sch
  • weather: clear

• 160215 22:30:12

  • cycleFridge enable=2, on all rx

• 160215 14:48:23 - 160215 22:30:12

  • /home/keck/sch/mce_autotune.sch
  • data/keckdaq/autoqueue/pending/7_ffflat_dslmast_32h_011.sch
  • weather: clearing, wind dying down, clouds on the horizon

• 160215 08:52:30

  • cycleFridge enable=2, on all rx

• 160215 01:09:49 - 160215 08:52:30

  • /home/keck/sch/mce_autotune.sch
  • /data/keckdaq/autoqueue/pending/7_ffflat_dslmast_31e_011.sch
  • weather: blowing snow

• 160214 20:07:40

  • cycleFridge enable=2, on all rx

• 160214 12:22:42 - 160214 20:07:40

  • /home/keck/sch/mce_autotune.sch
  • /data/keckdaq/autoqueue/pending/7_ffflat_dslmast_31c_011.sch
  • weather: hazy towards horizon, wind picking up, blowing snow

• 160214 07:15:57

  • cycleFridge enable=2, on all rx

• 160213 23:32:05 - 160214 07:15:56

  • /home/keck/sch/mce_autotune.sch
  • /data/keckdaq/autoqueue/pending/7_ffflat_dslmast_31j_011.sch
  • weather: clear, getting a bit hazy

• 160213 16:35:58

  • cycleFridge enable=2, on all rx

• 160213 08:54:10 - 160213 16:35:58

  • /home/keck/sch/mce_autotune.sch
  • /data/keckdaq/autoqueue/pending/7_ffflat_dslmast_32b_011.sch
  • weather: clear

• 160213 03:42:38

  • cycleFridge enable=2, on all rx

• 160212 19:45:09 - 160213 03:42:38

  • /home/keck/sch/mce_autotune.sch
  • /data/keckdaq/autoqueue/pending/7_ffflat_dslmast_31a_011.sch
  • weather: clear

• 160212 10:06:14

  • cycleFridge enable=2, on all rx

---

Winter 2016

---

• 160212 01:58:55 - 160212 10:06:14

  • /home/keck/sch/mce_autotune.sch
  • /data/keckdaq/autoqueue/pending/7_ffflat_dslmast_31a_011.sch
  • aborted schedule, it had a 2h wait from the day before in it
  • keck33 had problems, rebooted keck33
  • restarted gcp, masd
  • 160212 02:25:51 Starting schedule: /data/keckdaq/autoqueue/pending/7_ffflat_dslmast_31e_011.sch
  • weather: clear

• 160211 18:49:33

  • cycleFridge enable=2, on all rx

• 160211 16:19:19 - 160211 18:49:02

  • /data/keckdaq/autoqueue/pending/7_ffflat_dslmast_31a_011.sch
  • 160211 18:49:02 abort_schedule
  • finish work on membranes

• 160211 12:04:28

  • cycleFridge enable=2, on all rx

• 160211 04:24:24 - 160211 12:04:28

  • /home/keck/sch/mce_autotune.sch
  • /data/keckdaq/autoqueue/pending/7_ffflat_dslmast_31e_011.sch

• 160210 20:00:43 fridge cycle

  • 8_fridgecycle.sch
  • Double cycle on all rx

• 160210 09:00:15 - 160210 16:40:17 far field beam map

  • 7_ffflat_dslmast_31e_011.sch
  • dk=+166, rx4 under the mirror
  • 24″ chopper on DSL mast. 16 Hz chop rate. Chop reference in antenna0.pmac.fast_aux_input[0]
  • UC evap for rx0 ran out at ∼15:30:00
  • UC evap for rx3 ran out at ∼16:45:00
  • UC evap for rx1 ran out at ∼17:10:00
  • Ran out of disk space on keck32:/data during this schedule. Appears to have happened while writing file 20160210_143022.dat.gz

• 160209 02:49:07 - 160209 11:22:20 fridge cycle

  • cycleFridge enable=4, rx=rx0+rx1+rx2+rx3+rx4

• 160209 02:30:00

  • FFF is installed
  • changed compression to gzip for sat transfer

• 160208 21:00:00 far field flat install

  • Compressors off when we lowered elevation below 45 degrees.
  • Mirror installation went smoothly. el motor currents stayed within ±10 A during the lift.
  • Cooling back down. No need to pump on any receivers.
  • Moved mirror into back position for beam mapping. Final rod measurements:
    • A = 23.4375″
    • B = 23.625″
    • E = 57.5625″
    • F = 57.5″
    • G = 25.125″
    • H = 25.25″
  • Took some additional mirror measurements (see calibration hardware logbook).

• 160208 06:21:33 - 160208 12:14:09 1_knoise_responsivity_9_xxx.sch

  • Recycling voltage range used last year for 220GHz
  • added rx0 and rx2 back into set_default_hardware()

• 160207 20:51:02 - 160208 00:56:58 fridge cycle

  • cycleFridge enable=2, rx=rx2
  • Cycled rx2 only.

• 160206 16:24:51 - 160206 21:17:13 fridge cycle

  • cycleFridge enable=2, rx=rx0+rx1+rx3+rx4
  • rx2 is still too warm to cycle

• 160205 22:00:00 - 160206 03:00:00 load K1 and K3

  • Loaded receivers from MAPO into the Keck mount. First K1 (rx2) then K3 (rx0). Both receivers have been upgraded from 95 GHz to 220 GHz.
  • rx2 filled up with gas after warming up during the loading process. Pressure got as high as ∼1 mbar. Put cryostat on turbopump and pressure has come down steadily. Still need to assess whether it was a leak or just outgassing. Temperatures are much higher now in rx2 than other receivers. Will take a while to cool back down.