There has been significant research in adapting the Semantic Web technologies to create flexible context aware pervasive systems to enhance fields such as assisted living or smart environments. Several ontology based techniques have been proposed to simplify modeling knowledge and its relationships, and several ontology centered middleware tools are currently being developed to provide flexible and viable solutions for application developers. However, middleware built on the basis of Semantic Web generally suffers from drawbacks in performance, which limits its practical applications in the real world. This paper proposes a framework to facilitate Quality of Service (QoS) in ontology centered context aware pervasive middleware. Our approach suggests that context-aware middleware that operate by contracting mutual agreements with the client applications and provide controls over the amount of data to be processed by them can achieve predictable performance and response times. We also propose a service contract scheme that allows both client applications and middleware to participate in the decisions regarding the necessary data transformations required by the different system components in order to improve the overall system performance.