File API

Make sure to attend the Media Sprint Update Panel and join the Media Sprint 2009 at Open Media Camp in Denver, on April 18-19!

The presenters of this panel, including Aaron Winborn (aaron), Arthur Foelsche (arthurf), and Alex Urevick-Ackelsberg (Alex UA), will discuss the ongoing efforts of the Media Sprint by the Drupal Media working group to bring to fruition the Media module and related work.

This panel will begin with a demonstration and discussion of the current state of the ongoing development of the Media module, and a thorough examination of the concept for its extensible API. Finally, we'll discuss the near and far-term plans, including the Media Sprint planned for the next day of the Open Media Camp.

From my experience,there are a few major things lacking in Drupal. One of them that I have encountered a lot is the ability - or inability to manage files/documents effectively. I'd like to start laying the groundwork for a comprehensive file (or at least document) manager that allows some of the more popular feature requests e.g check in/check out, protected files & directories etc.

For the Media module, I'm trying to attach a new behavior to an AHAH form submit button. I need to open a previously hidden form element, BEFORE the ajax has loaded. This is ultimately for a throbber/progress indicator, and to allow the user to enter meta-data without having to wait for the 120MB video to finish uploading.

This will be a progress report for the ongoing Media Sprint, for the Media Module, including How You Can Get Involved™!

Note this immediately follows 8Tracks & Emfield.

Where
We're using Dojo's Dimdim - click this link at the time of the session.

When: Tuesday, March 24, 2009 at 4:00 PM (EST)

Modules for uploading and managing files, and associating them with content.

These modules actually have relatively little overlap, so you should consider this a guide in choosing the right module(s) for your needs, rather than a complaint about duplication. Just sayin'.

Contents

• What is the Media module?
• What is a stream?
• What is a wrapper?
• What is a PHP Stream?
• What is the Media project?
• Where can I find the Media project?
• Why do we need the Media project?
• Will the Media module support displaying multimedia files?
• What modules will support Media module?
• Who is working on the Media module?
• When can I use the Media module?
• What can I do to help?
• Is the Media module going to be in core?
• What functionality will the Media module support out of the box?

Anyone interested in creating a logo for the new Media group? Maybe Druplicon with a painter's smock and brush? Or a multi-color pixellated Druplicon? Or a video/image/music note montage swirling around Druplicon? Or ... ?

The following is the current list of Birds of a Feather (BOF) sessions on the white boards near the registration tables at Drupalcon DC that relate to Media. The list will certainly change over the week. Sessions relating to Media have been tagged on delicious. Please feel free to add those missing sessions! (This page is a wiki).

I wonder if something similar already exists:
I have a site with private download and organic groups. One client (i.e. one group) want's their files on their own server. The solution I'm thinking about is
1) User will upload files to example.com
2) That file will be sent to FTP sitting on the client's server.
3) Alter the links if the files and images to the FTP. Interesting enough, is that the <img> tag can point to an FTP.

• 'media mover' module can take care of the FTP part, although it might be an overkill for my needs.

The Media Sprint 2009 Wiki has a rough outline, based on an earlier document started by arthurf, but needs some serious fleshing out. I plan to begin working in earnest again on the sprint later next week and the following (am trying to slot 3-4 solid working days towards it).

There are also several volunteers who have offered to help over the coming weeks as well. We need to try to build some organization.

I am immensely interested in getting a rock-solid asset management system in place for Drupal 6, and my company is willing to pay for the work involved (my time spent). The purpose of this discussion is to 1) make sure that my assessment of the situation is accurate / common and 2) to make sure that whatever I come up with will be relatively forward-compatible with D7.

Media Code Sprint and Media Module Specifications

Table of contents

1. Introduction
2. Road map and Time Line
3. Use Cases
4. API Description
5. $item structure
6. Wireframes
7. Deliverables
8. Timeline
9. Risks
10. Resources

Introduction

I've recently committed the File (hook_file) module, which is an experimental backport of file.inc from Drupal 7 to Drupal 6. From the project page:

Per the great drewish's request... :) Some quick sample pseudo-code for how an SPL-based replacement for file_scan_directory() could work.

Note that this isn't tested, but is probably pretty close.

<?php
/**
* This pseudo-code neglects the following parameters:
* - $key: this can be handled a few different ways, but isn't super-important right now.
* - $callback: probably the most transparent way to handle this would be in an extension
* to the iterator we use.
* - $nomask: I can't remember off the top of my head if there's easy ways to do

(Note: I have moved the contents of this node to a sub-page of the http://drupal.org/community-initiatives/drupal-core section: http://drupal.org/node/361854 - webchick)

Currently, Embedded Media Field has Embedded Thumbnail, which allows editors to override the thumbnails provided by third party providers. I originally just copied some code from Image Field to handle it, after toying with (and rejecting) the idea of creating some sort of generic image handling api to help modules needing the functionality.

I've been doing some thinking about the future of the asset module. First of all, I'm completely dedicated to bringing it to Drupal 6 and beyond. What is not sure however is in which form it will move forward. With the improved hook_form and specifically with filefield and emfield tying into the youtube api, flvmediaplayer, the new media player et al, I certainly don't want to duplicate efforts. The asset module itself is quite a big project and maintaining it requires a lot of my time.

That's why I've been thinking lately of stripping the asset module down to the asset wizard: leaving the uploading of files to the *field modules (having the admin decide which modules he prefers and what type of files he needs) and managing them with the asset wizard. The asset wizard could be taken so much further if it could get all the attention it deserves. One thing I could use help on is the interface from a usability perspective: both on the asset wizard itself and on how to tie it in the different fields (filefield, emfield, ...) and editors (TinyMCE, WYSIWYG, ...). Everything is welcome: from concepts and wireframes to mock-ups and patches.

How do you want to handle the files and embedded media that have been already been imported by media mover, uploaded by filefield or integrated by emfield?

As announced, we've begun the Media Code Sprint to put better media handling into Drupal core!

It's been a great time so far. In the morning, drewish compiled a file function guessing game (free beer!), and andreiashu took the bait and completed the first round with great feedback about how unusable the current state is.

Next, we've sat down and begun writing SimpleTests for the hook_file patch. This has been great fun for me personally; though drewish is an old hand at building tests, this is completely new for me. And to top it off, webchick, Queen of Drupal testing, dropped by our table and got involved!

Even the process of writing tests has been helpful; we found at least one case for validation that had been missed in the original patch, and drewish decided that file_scan_directory needed refactoring. (He's currently chasing other problems as well, and cursing about finding himself going down rabbit holes.)

There's more fun to be had for all! If you want a quick 15 minute task that everyone is qualified to do, go play the File API Function Guessing Game. If you want to do more, then ping me (aaronwinborn), drewish, or dopry at #drupal in IRC, or leave a comment here!

(Cross-posted at AaronWinborn.com.)

We're running SimpleTests on various core modules, to get a baseline of what is fine, and what breaks with the hook_file patch.

I'm going to comment to this thread with results. The first run is on a clean installation of the build of Drupal 7.x-dev, Last updated: July 24, 2008 - 08:03.

Tests to run:

blog api
simpletest
system
upload
user
cache

As part of the Media/Files Code Sprint we're trying to re-evaluate the mess of badly named functions in file.inc. When we were discussing it on IRC, clouseau pointed out that it's been a popular drinking game at DrupalCon for a few years.

Take a look at each item in the list and make a guess about what you think each function does based solely on the name--to make this a fair fight I'm even giving you the parameter names--then look at the documentation and see how you did.

Please leave comments telling us which were the easiest to guess and which functions totally puzzled you. This is a big list so feel free to comment even if you don't make it all the way through.

The Media Code Sprint is underway! Here's a cross-post from my blog detailing the goals of this sprint, which runs through Saturday. We need your help!

Andrew Morton (drewish), Darrel O'Pry (dopry, remotely), and I are heading up a Media Code Sprint in Portland this week! Come help, in person or remotely, if you're interested in multimedia and Drupal! It has now officially started, and as I've volunteered to help keep folks updated, here goes...

First the reasons.

Number One: Better Media Handling in Core

Dries conducted a survey prior to his State of Drupal presentation at Boston Drupalcon 2008, and number one on the top ten (or 11) list of what would make THE KILLER DRUPAL 7 Release was "Better media handling".

Let me repeat that. Better media handling.

People have done really amazing stuff in contrib, but it is difficult (if not impossible in many cases) for developers to coordinate the use of files, as there is no good means for file handling in the core of Drupal. Thus, we have several dozen (or more) media modules doing some small part, or even duplicating functionality, sometimes out of necessity.

We need (better) media and file handling in Drupal core. In particular, there has been a patch for a hook_file in the queue for over a year, which has been in the Patch Spotlight (for the second time, no less) since May! (And has been RTBC several times during that process...) Come on folks.

One of the powers of Drupal is its system of hooks. We have hooks to modify nodes, to notify changes to user objects, to alter nearly any data (such as forms and menus). Noticeably absent is a consistent handling for files or any sort of notification. We need hook_file.

So goal Number One: get media handling in core. The means? Add hook_file and make files into a 1st class Drupal object. We'll be creating a test suite for functionality in the hook_file patch to validate it and "grease the wheels" to get it committed.

The other goals of this sprint pale in comparison to the first in utility, but are still highly desirable and worthwhile.

Number Two: Refactor File Functionality in Core

Hi,

Looking for suggestions for D5 non-node file handling, perhaps someone has done this before, perhaps there is a useful helper module.

So, basically I need my module to manage files without having a nid to attach it to. The files table has a nid column. Should I just manage my files separately from the files table?

Cheers
Simon

I'm declaring the week of OSCON as a media and files code sprint in Portland Oregon.

I would like to take an opportunity and introduce yet another file framework for Drupal 6. The project page is http://drupal.org/project/fileframework.

I would like to take an opportunity and introduce yet another file framework for Drupal 6. The project page is http://drupal.org/project/fileframework.

In order to draw more attention and reviews to patches that would improve the file support in Drupal 7 I've started the following list.

#247095 upload.module hard-codes 'view uploaded files' permission check
#203204 Uploaded Files have 600 permissions
#142995 Add hook_file and make files into a 1st class Drupal object

In order to draw more attention and reviews to patches that would improve the file support in Drupal 7 I've started the following list.

#247095 upload.module hard-codes 'view uploaded files' permission check
#203204 Uploaded Files have 600 permissions
#142995 Add hook_file and make files into a 1st class Drupal object

This article I wrote should appeal to people interested in high performance Drupal sites. Most of the performance gains are achieved by doing things with the Javascript and Drupal's File API.

http://wimleers.com/article/improving-drupals-page-loading-performance

You can expect another article soon, about the same topic, but then putting it to practice. I'll use the CDN integration module (of which I'm also the maintainer) and the included core patches to do it in just a few simple steps.

This article I wrote should appeal to people interested in high performance Drupal sites. Most of the performance gains are achieved by doing things with the Javascript and Drupal's File API.

http://wimleers.com/article/improving-drupals-page-loading-performance

You can expect another article soon, about the same topic, but then putting it to practice. I'll use the CDN integration module (of which I'm also the maintainer) and the included core patches to do it in just a few simple steps.

This week I "got around to it". That thing I knew I should do for so long, but never actually wanted to do. Well this week the itch could not be ignored. Finally I scratched. And now I share with you...

UpAPI

I thought of calling it "Yet Still One More File Helper Module," but decided shorter was better. Because shorter is what this module will do for you, if you need to add file uploads to your module, that is. Your code will be shorter and your time spent will be shorter. At least that's the goal, and you can help by playing with this new module and giving some feedback.

This week I "got around to it". That thing I knew I should do for so long, but never actually wanted to do. Well this week the itch could not be ignored. Finally I scratched. And now I share with you...

UpAPI

I thought of calling it "Yet Still One More File Helper Module," but decided shorter was better. Because shorter is what this module will do for you, if you need to add file uploads to your module, that is. Your code will be shorter and your time spent will be shorter. At least that's the goal, and you can help by playing with this new module and giving some feedback.

Please help test the fine patch at http://drupal.org/node/115267

Please help test the fine patch at http://drupal.org/node/115267

</style></head><body dir="ltr"><p class="P1">Application for Summer of Code 2007:  MultiPublisher Module</p><p class="P1">   by Ryan Grace  <a href="http://www.thefractal.org/">www.thefractal.org</a> :: ryan@thefractal.org </p><p class="P2"> </p><p class="P2">Synopsis </p>

</style></head><body dir="ltr"><p class="P1">Application for Summer of Code 2007:  MultiPublisher Module</p><p class="P1">   by Ryan Grace  <a href="http://www.thefractal.org/">www.thefractal.org</a> :: ryan@thefractal.org </p><p class="P2"> </p><p class="P2">Synopsis </p>

Greetings! I don't intend for this to be an advertisement for Amazon, so if folks want to discuss other similar services that's totally cool, and we might even change the name of the group. But, I've been spending (too many) hours researching Amazon Web Services recently and thought some people might be interested to know what it's all about and how they might be able to use this stuff...

Greetings! I don't intend for this to be an advertisement for Amazon, so if folks want to discuss other similar services that's totally cool, and we might even change the name of the group. But, I've been spending (too many) hours researching Amazon Web Services recently and thought some people might be interested to know what it's all about and how they might be able to use this stuff...

Hello, I've recently submitted this patch for the backup module which allows automated regular backups of a Drupal installation to Amazon S3. It currently uses Geoffrey Gaudreault's Amazon S3 PHP class, which requires the Crypt_HMAC and (hacked) HTTP_Request PEAR modules, as well as forcing the site developer to alter some variables within the code of the class. I'm hoping the File API will offer a much easier way to integrate the backup.module with Amazon S3.

Hello, I've recently submitted this patch for the backup module which allows automated regular backups of a Drupal installation to Amazon S3. It currently uses Geoffrey Gaudreault's Amazon S3 PHP class, which requires the Crypt_HMAC and (hacked) HTTP_Request PEAR modules, as well as forcing the site developer to alter some variables within the code of the class. I'm hoping the File API will offer a much easier way to integrate the backup.module with Amazon S3.

So there is quite a bit of duplicated code and convoluted interfaces for manipulating files. especially images...

imagecache, image, watermark, imageexact, etc, etc are all relatively simple 'filters' or 'processors' for files. I've got different versions of imagecache with various gd filters, image2ascii conversion, mosaic affects, etc floating around, along with some tools for processing quicktime files and front-ends for ffmpeg I'd like to make available to the community, but coding a module for each one would be a real pain, and would make for a rather module heavy drupal...

imagecache is really a proof of concept work for generating images on the fly, but I had to throw the 'preset' and 'action' management interface into it so it could do its good work. I think there is a strong need to sort this kind of processing into its own little library. Actions.module or filter.module would be a good model for a UI I think.

So there is quite a bit of duplicated code and convoluted interfaces for manipulating files. especially images...

imagecache, image, watermark, imageexact, etc, etc are all relatively simple 'filters' or 'processors' for files. I've got different versions of imagecache with various gd filters, image2ascii conversion, mosaic affects, etc floating around, along with some tools for processing quicktime files and front-ends for ffmpeg I'd like to make available to the community, but coding a module for each one would be a real pain, and would make for a rather module heavy drupal...

imagecache is really a proof of concept work for generating images on the fly, but I had to throw the 'preset' and 'action' management interface into it so it could do its good work. I think there is a strong need to sort this kind of processing into its own little library. Actions.module or filter.module would be a good model for a UI I think.

Sorry, I would've posted this to the What do you want in a file API? thread, but unfortunately you can't attach files to comments.

These are some use cases that Shane Peery, one of our clients, wrote up for AMBuzz, an intranet site for a large media distribution company. They have some fairly complex needs, so I figured I'd post them here in the hopes they might inspire some of the folks working on this stuff, and also in case anyone had any bright implementation ideas. :)

I've also attached some general observations/comments he made regarding Drupal's file API as someone with experience in programming, but new to Drupal. Some of the stuff he mentions is relatively easy to sort out (for example, the security stuff is a matter of turning on private downloads and moving the files directory outside of public_html), but I think there are some interesting ideas here anyway.

Sorry, I would've posted this to the What do you want in a file API? thread, but unfortunately you can't attach files to comments.

These are some use cases that Shane Peery, one of our clients, wrote up for AMBuzz, an intranet site for a large media distribution company. They have some fairly complex needs, so I figured I'd post them here in the hopes they might inspire some of the folks working on this stuff, and also in case anyone had any bright implementation ideas. :)

I've also attached some general observations/comments he made regarding Drupal's file API as someone with experience in programming, but new to Drupal. Some of the stuff he mentions is relatively easy to sort out (for example, the security stuff is a matter of turning on private downloads and moving the files directory outside of public_html), but I think there are some interesting ideas here anyway.

I'm cross posting an issue here to let you guys know of this way:

Private and Public File Access

Banner.module isn't really guilty here. There are a lot more modules that do not work with private file access.

I'm cross posting an issue here to let you guys know of this way:

Private and Public File Access

Banner.module isn't really guilty here. There are a lot more modules that do not work with private file access.

Hey there - I work for the Electronic Frontier Foundation (EFF), and we are migrating our huge old legacy site into Drupal. We are trying to get some help developing some additions to the file upload module, and Zack suggested I post our specs here for feedback from you all.

Here's the email I sent him with the specs of our dream file upload module:

EFF needs a file upload and organization module that interacts with files stored on the filesystem. There are some contributed modules with similar functionality, but none that do exactly what we need. It's my belief that adding functionality to the existing 'upload' module that comes with Drupal would be a fine approach.

Hey there - I work for the Electronic Frontier Foundation (EFF), and we are migrating our huge old legacy site into Drupal. We are trying to get some help developing some additions to the file upload module, and Zack suggested I post our specs here for feedback from you all.

Here's the email I sent him with the specs of our dream file upload module:

EFF needs a file upload and organization module that interacts with files stored on the filesystem. There are some contributed modules with similar functionality, but none that do exactly what we need. It's my belief that adding functionality to the existing 'upload' module that comes with Drupal would be a fine approach.

The handling and publishing of files is one of Drupals weakest points. My site was largely based on me publishing mp3 files and code. I wanted to upload the files with FTP and publish them from an upload directory on the server. At that point I made my own "filepublishing module". And from that point of I started to ponder on what could be done with the way Drupal handles files. And now I have a rough idea of how I would like to do it.
I probably won't have the time to do any time soon, so I'm sharing it here to make shure the idea is not lost when I start to think about other things.

The handling and publishing of files is one of Drupals weakest points. My site was largely based on me publishing mp3 files and code. I wanted to upload the files with FTP and publish them from an upload directory on the server. At that point I made my own "filepublishing module". And from that point of I started to ponder on what could be done with the way Drupal handles files. And now I have a rough idea of how I would like to do it.
I probably won't have the time to do any time soon, so I'm sharing it here to make shure the idea is not lost when I start to think about other things.

In another comment (Multisite with separate files/ per vhost) I discuss a problem that I am having with moving databases between multisite installations when they each have their own files/ directory. It concludes:

A solution is to not store the filepath prepended to every filename in files.filepath, but rather to prepend the filepath when needed. Not sure where to begin (maybe file_create_path) but I thought I'd float the idea to see what others think of it before I actually attempt to patch core - and probably some modules...