A look back on the challenges and opportunities of upgrading the Blackboard VLE when self-hosted.

1. Upgrade Club
“Lightning Talk”

2. Agenda
About upgrade club on the community site
Our practice and experiences at Southampton
Open discussion

3. Upgrade Club
• Started 2016 having been given 3 months to arrange an upgrade.
• 2016 thread had 193 replies
• 2017 thread had 349 replies
• 2018 thread has 56 replies so far

4. Upgrade Club
• Threads are full of useful tips, questions, groans of disappointment and exaltations of
success
• Members have a wide variety of backgrounds and skills
• Very positive, open, collaborative environment

5. Questions
• Have you posted in Upgrade Club?
• Are you upgrading Blackboard this year?
– To what version? 2016 Q4, 2017 Q2, 2017 Q4,
2018 Q2?
• Are you self-hosted / managed hosted / SAAS?

6. Southampton Upgrade History
• Started with Blackboard 5.0 in 2000
• 16 Major upgrades since then.
• Did you know?
– You can review your upgrade history by going to
• System admin 🡺 System configuration 🡺 System Information

10. Southampton Bb Upgrades
• Working in a large IT department
• Prince 2, ITIL, Lean Six Sigma
• Very hierarchical and structured
• If you can imagine it - there is a board you have to go to discuss it
• Upgrades done out of hours = overtime = requires money = requires more project
documentation

11. IT dept organogram
• ~375 staff
• Bb upgrades involve staff from
multiple teams

12. Bb upgrade projects at Southampton
Project elements
• Project Brief
• Business Case
• Monthly Highlight
Reports and
Directorate Review
• Project Tasks for
almost every activity
(about 50 tasks for
an upgrade)
• Arrange downtime
Upgrade work
• Upgrades to new
release in devs and
preprod
• Testing
• Disaster Recovery
• Upgrades to latest
CU
• Documentation
(internal and end-
user)
Go live
• Change
Management
Approval
• Communications
• Weekend Upgrade
• Monitoring
Closure
• Lessons learned
• End project report
• Benefits realization
review
• Update service
roadmap
• Prepare for next
project

13. Questions
• Do you run Blackboard upgrades as projects / using
project management methodology?
• Do you do Blackboard upgrades out of hours?
• Is it simple to arrange budget for overtime / TOIL?

14. Working on Bb upgrades: lessons, recommendations, experiences
• A post giving an overview is at https://bit.ly/2HaDt1c (from Upgrade Cohort 2018 community area)
• Upgrade club blog: https://bit.ly/2qgL7MY
• The next slides are some of my “highlights”

15. Nothing here is “special”
• Every institution is different
• Nothing in this presentation is out of the
ordinary
• Make sure to see Jonathan Knight’s
presentation “Early Adoption and Minimal
Testing”

16. Building the business case
To get budget to pay for overtime, need a full project
business case.
This includes such items as:
Reasons to upgrade, benefits, resource requirements
and costs, risks, etc..
Business case for our 2018 Blackboard upgrade was 37
pages long

17. Business Case tips
• Reasons
• http://library.blackboard.com/docs/support/Blackboard_Learn_Support_Services_Guide.pdf

18. Business Case tips
• Benefits
• Screen grab roadmap webinars to use as evidence of benefits from new versions

19. Question
• Do you perform a benefits realisation analysis
following upgrades?

20. Implementation Plan
Start building as soon as you can.
I find using a wiki very useful. It’s quick to edit and can
be structured so that plans can be copied easily and
elements edited
Having a great plan that can be updated each year will
save time.

21. Structure of a typical upgrade plan
1. Prepare files (installer, installer properties, back up
files that installer deletes)
2. Oracle patching
3. Upgrade Blackboard
4. Review configuration changes
5. Make configuration changes
6. Pushconfigupdates
7. GUI based config and testing
8. Removal of temporary files (installer, backed up files
etc.)

22. Verification scripts
Installer will wipe your carefully set fixes,
workarounds, optimizations.
This year we started building verification scripts to
quickly identify whether settings needed to be reset
Repeat verification scripts after doing
pushconfigupdates

23. Protip – use VMware Snapshots
• We take a VMware “cold snapshot” of
our vApp after each upgrade stage.
• If something goes wrong we can restore
environment back to how it was within
15 minutes.

24. Keep up to date with issues / recommendations
From the community
• Mailing lists still have the
best info (e.g. ASU BB-
ADMIN-L)
• Community site has lots of
useful info and a good
place to ask questions
• BB World / BB TLC / User
groups / DevCon
From Blackboard
• Generate known issues
lists from support site
• Subscribe to all support
notifications
• Bb support will often give
extra help
• Build a good
relationship, complete
support surveys
From within
• Make your plans open
within your department
• Encourage feedback and
ideas
• Share lessons, build an
environment of
collaboration

25. Bb notifications

26. Question
• Do you have any other tips on keeping up to date
with known issues?

27. Upgrade frustrations
• Installer will overwrite fixes Blackboard
support asked you to implement to resolve
known issues.
• Some are fixes are more than 4 years
old (e.g. 000039703, 000037634)
• So you have to implement them again.
• For settings within bb-config.properties you
can set most of these in the
installer.properties file.

28. Additional settings we are using in the installer.properties files
• bbconfig.jvm.options.extra.tomcat
• bbconfig.jvm.options.gc
• bbconfig.email.use.dmarc.from.override
• bbconfig.max.stacksize.tomcat
• bbconfig.appserver.http.compression
• bbconfig.jvm.options.codecache.reserved
• bbconfig.jvm.options.codecache.initial
• bbconfig.cs.database.maxpoolsize
• bbconfig.peer.discovery.timeout.inactive
• bbconfig.peer.discovery.timeout.dead
• bbconfig.server.backend.processor
• bbconfig.gradecenter.cache.grade_threshol
d

29. Document “fixes” separately
• Those key fixes and workarounds can get “lost”
in implementation plans.
• I found some fixes we implemented in 2014
had been lost in our 2016 upgrade because no
one was left who knew about them.
• Keeping a separate list of fixes that should be
re-applied until they are resolved centrally
should save time and ensure they are not
“lost”.
• Ours is now 17 pages long (22 fixes)

30. My “favourite” fixes so far
Installer failed for no apparent reason. Cause: random
number generator not random enough (Thanks to
Cherif Abbes /Bb for the fix)
SCORM disconnection fix (fix was to update click
jacking settings) (Thanks to Stuart Robinson and the
team at Leeds for the fix)
High CPU / Load caused by
MicrosoftDocumentParser.sh (Thanks to Chris Filkins
for the fix)

31. More Upgrade frustrations
• Upgrades that remove functionality without
replacing it
• E.g.
• Virtual classroom and chat
• Crocodoc functionality loss
• Upgrades that add functionality which is broken
• E.g. availability toggle in 2017 Q4

32. Question
• What are your upgrade frustrations?

33. Testing
• We test core functionality and integrations.
• We accept we can’t test everything and rely on
– Bb support notifications
– Mailing List / Community site
– Amy Eyre from York for tipping us off about new
issues
• We also perform a disaster recovery exercise and
a load testing exercise.

34. Communications during upgrade
Outage page with embedded twitter feed
Keep an updated ETA completion
Visual indicator of progress (pie chart)

35. Celebrating success
Celebratory fried breakfast paid out of project budget
(but not allowed to do this any longer ☹)
Ensure overtime payments / TOIL processed quickly
Arrange “thank you” email from University executive

36. Summary
Prepare
While onerous, building methodical project
documentation is helpful in the long-term and
often a requirement for funding.
Research
Get on the mailing lists, subscribe to Bb
notifications, use the community, contribute
to upgrade club ☺. Be nice to Bb support!
Upgrade
Careful documentation and verification
essential (before we do our live upgrade we
will have practiced it six times already).
Celebrate and learn
Celebrate success and note lessons and
recommendations for next time.

37. Questions and discussion