The slides for the talk I gave at SymfonyCon Lisbon 2018

1. Olá SymfonyConOlá SymfonyCon
Lisboa!Lisboa!

2. File Storage forFile Storage for
ModernModern PHPPHP ApplicationsApplications

3. Hi, I'm Frank!Hi, I'm Frank!
@frankdejonge@frankdejonge

4. Free-lance DeveloperFree-lance Developer
++
Free-software DeveloperFree-software Developer

5. I18n Routing in 4.1I18n Routing in 4.1
Routing Speedup in 3.2Routing Speedup in 3.2

6. EventSauce.ioEventSauce.io
Event-sourcing LibraryEvent-sourcing Library

7. FlysystemFlysystem
Filesystem AbstractionFilesystem Abstraction

8. 50.000.000 installs!50.000.000 installs!
FlysystemFlysystem

11. The things I've seen...The things I've seen...The things I've seen...The things I've seen...The things I've seen...

13. FTP FTP

14. FTP = FTP =

15. File storage forFile storage for modernmodern
PHP applications.PHP applications.

16. It depends!It depends!
(and that's the problem)(and that's the problem)

17. 1. Filesystems
2. Modern Apps & FS
3. Choosing an FS
4. Storage in Code

18. How not to fail atHow not to fail at
filesystemsfilesystems..

19. What is aWhat is a
filesystem?filesystem?

21. FILESYSTEMS EVERYWHEREFILESYSTEMS EVERYWHEREFILESYSTEMS EVERYWHEREFILESYSTEMS EVERYWHEREFILESYSTEMS EVERYWHERE

24. Common ProblemsCommon Problems
and/or Symptomsand/or Symptoms

25. Unstructured FilesystemUnstructured Filesystem
The "/uploads/" directory.The "/uploads/" directory.

26. Are these files used?Are these files used?

27. Can I delete these files?Can I delete these files?

28. Where did these filesWhere did these files
come from?come from?

30. Can we changeCan we change
filesystems?filesystems?

32. But is it fine?But is it fine?But is it fine?But is it fine?But is it fine?

33. Assumptions
Creating a feature

34. Assumptions
Creating a feature
Experience with tools
Feature created

35. Assumptions
Creating a feature
Experience with tools
Feature created
Initial usage
Put in production

36. Assumptions
Creating a feature
Experience with tools
Feature created
Initial usage
Put in production
Actual Usage
Feature Adoption

37. Assumptions
Creating a feature
Experience with tools
Feature created
Initial usage
Put in production
Actual Usage
Feature Adoption
Experience
Actual Problems

39. Assumptions
Creating a feature
Experience with tools
Feature created
Initial usage
Put in production
Actual Usage
Feature Adoption
Experience
Actual Problems

41. What is aWhat is a
filesystem, actually?filesystem, actually?

42. A filesystem is a
system that controls
how data is stored
and retrieved.

43. A system is a
collection of parts,
conventions, and
interactions, forming
a complex whole.

44. So, what is aSo, what is a
filesystem?filesystem?

45. A filesystem is a system
that provides a collection
of parts, conventions, and
interactions, forming a
complex whole that
controls how data is
stored and retrieved.

47. Something you use toSomething you use to
store and retrieve data.store and retrieve data.

48. src/
Filesystem.php
AdapterInterface.php
tests/
Filesystem.php
AdapterInterface.php
src/Filesystem.php
src/AdapterInterface.php
tests/Filesystem.php
tests/AdapterInterface.php
Structure types:Structure types:
Nested or Linear

49. Nested filesystemsNested filesystems
Are very common.
Your desktop has it.
Have directories
which have files and more directories.

50. Linear filesystemLinear filesystem
Are like key/value stores
only have files
require a different approach.
are very "cloud"

51. Filesystem OfferingFilesystem Offering

52. FS = NASFS = NAS
AWS Volume |AWS Volume | Digital Ocean BlobDigital Ocean Blob
StorageStorage | Gluster FS || Gluster FS | CephCeph

53. FS + CDNFS + CDN
AWS S3 |AWS S3 | Digital Ocean SpacesDigital Ocean Spaces
Rackspace Cloud Files |Rackspace Cloud Files | GoogleGoogle
Cloud StorageCloud Storage

54. FS + SharingFS + Sharing
Dropbox |Dropbox | Box.comBox.com || Google DriveGoogle Drive
Microsoft OneDriveMicrosoft OneDrive

55. FS + Not Really a FSFS + Not Really a FS
WebDAV |WebDAV | MongoDB GridFSMongoDB GridFS
PostgreSQL Large Object StoragePostgreSQL Large Object Storage

56. What isWhat is NOTNOT
a filesystem?a filesystem?

57. Content Distribution NetworkContent Distribution Network
Streaming ServerStreaming Server
File Sharing ServiceFile Sharing Service
Media ServerMedia Server
File ManagerFile Manager
Web ServerWeb Server
DatabaseDatabase

59. Ryan Singer (2010)Ryan Singer (2010)
So much complexity
in software comes
from trying to make
one thing do two
things.

60. Frank de Jonge (right now)Frank de Jonge (right now)
Stop trying to make a
filesystem do 100
things.

61. Create
Read
Update
Delete

62. Create
Read
Update
Delete
Inspect
List

63. CRUDCRUDLILI™™

64. Filesystems and PHP.Filesystems and PHP.

65. $result = file_put_contents($location, $contents);
$contents = file_get_contents($location);
unlink($location);
$timestamp = mtime($location);
$mimetype = (new Finfo(FILEINFO_MIME_TYPE))->file($location);

66. glob($location . '/*'); // ?
scandir($location); // ?
readdir(opendir($location));
// Or...
array_map('normalize_fileinfo',
iterator_to_array(new DirectoryIterator($location)));
// Or...
array_map('normalize_fileinfo',
iterator_to_array(new RecursiveIteratorIterator(
new RecursiveDirectoryIterator($location))));

67. WATWATWATWATWAT

68. $resource = fopen($src, 'r+');
$handle = fopen($dest, 'w+');
while ( ! feof($resource)) {
$chunk = fread($resource, 1024);
fwrite($handle, $chunk, 1024);
}
$result = fclose($handle);
fclose($resource);

69. $resource = fopen($src, 'r+');
if ($resource === false) handle_error(); //
$handle = fopen($dest, 'w+');
if ($handle === false) handle_error(); //
while ( ! feof($resource)) {
$chunk = fread($resource, 1024);
if ($chunk === false) handle_error(); //
$writeResult = fwrite($handle, $chunk, 1024);
if ($writeResult === false) handle_error(); //
}
$result = fclose($handle);
if ($result === false) handle_error(); //
$closeResult = fclose($resource);
if ($closeResult === false) handle_error(); //

70. WATWATWATWATWAT

71. /**
* Why not just use .... ?
*/
copy($src, $dest);

72. use Flysystemuse Flysystem
for file storagefor file storage

73. Uniform APIUniform API

74. use LeagueFlysystemAdapterLocal;
$adapter = new Local(__DIR__.'/path/to/dir');
$filesystem = new Filesystem($adapter);
$filesystem->write($destination, $contents);
$filesystem->read($location);
$filesystem->update($destination, $contents);
$filesystem->listContents($location, $recursive);
$filesystem->delete($location);
$filesystem->getSize($location);

75. use LeagueFlysystemAwsS3v2AwsS3Adapter;
$adapter = new AwsS3Adapter($client, 'bucket-name');
$filesystem = new Filesystem($adapter);
$filesystem->write($destination, $contents);
$filesystem->read($location);
$filesystem->update($destination, $contents);
$filesystem->listContents($location, $recursive);
$filesystem->delete($location);
$filesystem->getSize($location);

76. use LeagueFlysystemAdapterFtp;
$adapter = new Ftp($ftpSettings);
$filesystem = new Filesystem($adapter);
$filesystem->write($destination, $contents);
$filesystem->read($location);
$filesystem->update($destination, $contents);
$filesystem->listContents($location, $recursive);
$filesystem->delete($location);
$filesystem->getSize($location);

77. $resource = $remoteFs->readStream('some/path.mp4');
$localFs->writeStream('another/path.mp4', $resource);
fclose($resource);

78. $resource = $remoteFs->readStream('some/path.mp4');
$localFs->writeStream('another/path.mp4', $resource);
fclose($resource);
// do stuff
$resource = $localFs->readStream('another/path.mp4');
$remoteFs->writeStream('some/path.mp4', $resource);
fclose($resource);

79. $resource = $azure->readStream('some/path.mp4');
$aws->writeStream('another/path.mp4', $resource);
fclose($resource);

80. Relative PathsRelative Paths

81. domain/concern.txt
domain/concern.txt
/configuration/
/new/path/

82. use LeagueFlysystemAdapterLocal;
// Configuration
$adapter = new Local('/configuration/');
$filesystem = new Filesystem($adapter);
// Usage
$filesystem->writeStream('financial-reports/latest.csv', $resource);

83. use LeagueFlysystemAdapterLocal;
// Configuration
$adapter = new Local('/new/path/');
$filesystem = new Filesystem($adapter);
// Usage
$filesystem->writeStream('financial-reports/latest.csv', $resource);

84. use LeagueFlysystemAwsS3v2AwsS3Adapter;
// Configuration
$adapter = new AwsS3Adapter($client, 'bucket-name');
$filesystem = new Filesystem($adapter);
// Usage
$filesystem->writeStream('financial-reports/latest.csv', $resource);

85. Flysystem =Flysystem =
No Vendor Lock-inNo Vendor Lock-in
Isolation +Isolation +
Encapsulation +Encapsulation +
Predictability +Predictability +
PortabilityPortability

86. Don't use Flysystem for everything.Don't use Flysystem for everything.

87. Choosing anotherChoosing another
FilesystemFilesystem

92. Using the localUsing the local
filesystem isfilesystem is

101. Why?Why?
We want to scale (horizontally).
... so, we want stateless apps.
We don't have to serve files ourselves.
Share files across multiple application.

102. We're not askingWe're not asking
the rightthe right questionsquestions..

103. Where?Where?

104. On Premise?On Premise?On Premise?On Premise?On Premise?

105. "Cloud""Cloud""Cloud""Cloud""Cloud"
Still a Computer™Still a Computer™Still a Computer™Still a Computer™Still a Computer™

106. Who will doWho will do
maintenance?maintenance?

107. How will weHow will we
use the files?use the files?

108. Why are weWhy are we
going to store it?going to store it?

109. How long are weHow long are we
going to keep the files?going to keep the files?

110. How big willHow big will
the files be?the files be?

111. Needs > CapabilitiesNeeds > Capabilities

112. NeedNeed
Something you requireSomething you require

113. CapabilityCapability
What something can doWhat something can do

114. Write Down!Write Down!

115. - Agile Manifesto- Agile Manifesto
Working Software
over Comprehensive
Documentation

117. + +

118. ADRADR
Architecture Decision RecordArchitecture Decision Record

119. ContextContext
The team uses AWS / AzureThe team uses AWS / Azure

120. ContextContext
We expect massive storage growthWe expect massive storage growth

121. RequirementsRequirements
Data must be stored on premiseData must be stored on premise

122. RequirementsRequirements
Data must be stored in the EUData must be stored in the EU

123. DecisionDecision
We'll use AWS S3 / Azure BlobWe'll use AWS S3 / Azure Blob
StorageStorage

124. ConsequencesConsequences
Operations of files will requireOperations of files will require
download and re-upload.download and re-upload.

125. Share ResponsibilitiesShare Responsibilities

126. Assumptions
Creating a feature
Experience with tools
Feature created
Initial usage
Put in production
Actual Usage
Feature Adoption
Experience
Actual Problems

127. Create Use-Case
Driven Storage
Models

128. interface ProfilePictureStorage {
/**
* @param resource $profilePicture
*/
public function store(User $user, $profilePicture);
}

129. interface FinancialReportStorage {
/**
* @param resource $reportFile
*/
public function store(Organization $organization, $reportFile);
}

130. class FilesystemFinancialReportStorage
implements FinancialReportStorage {
private $filesystem;
public function __construct(FilesystemInterface $filesystem) {
$this->filesystem = $filesystem;
}
public function store(Organization $organization, $reportFile)
{
$path = sprintf('%s/%s/financial-report.csv',
$organization->id()->toString(),
date('Y/m/d'));
$this->filesystem->writeStream($path, $reportFile);
}
}

131. Isolate your use-case.Isolate your use-case.

132. Assumptions
Creating a feature
Experience with tools
Feature created
Initial usage
Put in production
Actual Usage
Feature Adoption
Experience
Actual Problems

133. Document experiencesDocument experiences
When stuff goesWhen stuff goes

134. (Blameless) Post-Mortem(Blameless) Post-Mortem

135. Storage was unavailableStorage was unavailable
Full diskFull disk
Deleted old filesDeleted old files
manuallymanually
Automated cleanupAutomated cleanup
routinesroutines
# What happened# What happened
# What caused it# What caused it
# What fixed it# What fixed it
# How to prevent# How to prevent

136. ResuméResumé

137. Use FlysystemUse Flysystem

138. Keep it simpleKeep it simple

139. Documents requirementsDocuments requirements

140. Document decisionsDocument decisions
(using ADR)(using ADR)

141. CreateCreate specificspecific storagestorage
classes per use-caseclasses per use-case

142. Use generic solutions forUse generic solutions for
generic problemsgeneric problems

143. Use special solutions forUse special solutions for
special problemsspecial problems

146. Gluster FSGluster FS

147. Gluster FSGluster FS
on Kuberneteson Kubernetes

148. Persistent VolumePersistent Volume
on Gluster FSon Gluster FS
on Kuberneteson Kubernetes

149. AA
ServiceService
on Kuberneteson Kubernetes
with PersistentVolumewith PersistentVolume
on Gluster FSon Gluster FS
on Kuberneteson Kubernetes

150. So what do I do?So what do I do?

151. It depends!It depends!

152. Thank you for listening!Thank you for listening!
@frankdejonge@frankdejonge